Oct 30 19:15:10.426841 ip-10-0-133-7 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:15:10.807903 ip-10-0-133-7 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:10.807903 ip-10-0-133-7 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:15:10.807903 ip-10-0-133-7 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:10.808455 ip-10-0-133-7 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:10.812404 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812317    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:10.812404 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812405    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812409    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812413    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812415    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812418    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812421    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812424    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812426    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812429    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812431    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812434    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812436    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812439    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812441    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812444    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812446    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812448    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812451    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812454    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:10.812466 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812457    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812459    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812462    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812464    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812467    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812471    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812476    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812480    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812484    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812487    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812490    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812492    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812496    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812500    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812503    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812506    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812508    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812511    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812515    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:10.812913 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812517    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812520    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812523    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812525    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812528    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812530    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812545    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812548    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812551    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812553    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812556    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812558    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812561    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812563    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812565    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812570    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812573    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812575    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812578    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812581    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:10.813343 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812583    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812586    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812589    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812591    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812594    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812596    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812599    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812601    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812604    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812606    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812609    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812612    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812615    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812617    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812620    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812622    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812626    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812628    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812631    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812634    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:10.813799 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812637    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812639    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812641    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812644    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812647    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812649    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812652    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812655    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812658    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812660    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812663    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812665    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812668    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812670    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812673    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812675    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812678    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812681    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812683    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812686    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:10.814277 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812688    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812691    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812713    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812717    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812721    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812724    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.812726    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813530    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813551    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813554    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813558    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813560    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813563    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813565    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813568    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813570    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813573    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813577    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813579    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:10.814732 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813582    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813585    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813588    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813590    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813593    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813595    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813598    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813600    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813605    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813609    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813612    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813614    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813617    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813619    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813622    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813624    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813626    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813629    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813632    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:10.815193 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813635    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813637    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813640    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813642    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813645    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813648    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813650    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813653    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813655    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813657    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813660    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813663    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813666    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813669    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813671    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813674    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813676    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813679    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813681    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813684    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:10.815682 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813686    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813689    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813691    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813693    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813696    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813699    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813701    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813703    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813706    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813708    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813711    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813715    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813719    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813722    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813724    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813727    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813730    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813732    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813734    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:10.816246 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813737    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813739    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813742    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813744    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813747    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813749    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813752    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813754    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813757    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813760    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813762    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813764    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813767    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813769    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813772    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813775    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813777    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813779    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813782    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813784    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:10.816701 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813787    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813789    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813791    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813794    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813796    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813799    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813802    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813805    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813808    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813811    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813813    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813816    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813818    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813821    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813823    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.813826    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814299    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814304    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814307    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814310    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:10.817273 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814312    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814315    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814317    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814321    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814323    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814326    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814328    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814331    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814333    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814336    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814338    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814341    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814344    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814346    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814349    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814352    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814354    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814357    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814360    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:10.817752 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814362    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814365    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814368    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814373    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814375    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814378    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814380    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814383    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814385    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814387    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814390    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814392    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814395    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814397    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814400    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814402    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814406    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814409    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814412    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814415    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:10.818171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814418    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814420    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814423    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814426    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814428    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814431    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814435    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814438    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814442    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814444    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814447    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814449    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814453    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814455    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814459    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814463    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814466    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814468    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814470    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:10.818637 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814473    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814475    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814478    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814480    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814483    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814485    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814488    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814490    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814493    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814495    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814497    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814500    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814502    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814505    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814507    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814509    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814512    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814514    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814517    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814519    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:10.819072 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814522    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814524    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814527    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814529    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814531    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814549    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814552    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814557    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814560    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814563    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814565    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814568    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814570    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814573    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814575    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814578    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814580    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814582    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814585    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:10.819515 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814587    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814590    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814592    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814595    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.814597    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814697    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814707    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814717    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814724    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814729    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814732    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814737    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814741    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814744    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814747    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814750    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814754    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814757    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814759    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814762    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814768    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814772    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814775    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:15:10.819954 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814778    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814781    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814784    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814787    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814791    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814794    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814797    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814800    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814803    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814806    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814809    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814811    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814815    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814817    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814822    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814825    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814827    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814830    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814833    2579 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814836    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814840    2579 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814843    2579 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814846    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814849    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:15:10.820466 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814852    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814856    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814859    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814862    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814865    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814870    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814874    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814877    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814880    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814883    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814885    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814888    2579 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814892    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814895    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814898    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814901    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814904    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814907    2579 flags.go:64] FLAG: --help="false"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814910    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814913    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814916    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814919    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814922    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:15:10.821086 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814925    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814929    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814931    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814934    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814937    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814940    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814943    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814945    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814949    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814952    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814954    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814957    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814961    2579 flags.go:64] FLAG: --lock-file=""
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814964    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814969    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814972    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814978    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814981    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814984    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814987    2579 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814990    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814993    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814996    2579 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.814998    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815003    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:15:10.821627 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815006    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815009    2579 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815012    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815015    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815018    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815021    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815024    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815027    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815030    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815037    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815039    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815042    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815046    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815048    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815051    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815055    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815057    2579 flags.go:64] FLAG: --port="10250"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815060    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815063    2579 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-006473e1ec172c070"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815066    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815069    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815073    2579 flags.go:64] FLAG: --register-node="true"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815078    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815082    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815085    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815088    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:15:10.822191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815091    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815095    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815098    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815101    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815104    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815107    2579 flags.go:64] FLAG: --runonce="false"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815109    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815112    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815115    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815118    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815121    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815124    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815127    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815129    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815132    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815135    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815138    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815140    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815143    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815146    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815149    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815154    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815157    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815160    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815164    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:15:10.822851 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815166    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815170    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815174    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815177    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815181    2579 flags.go:64] FLAG: --v="2"
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815185    2579 flags.go:64] FLAG: --version="false"
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815189    2579 flags.go:64] FLAG: --vmodule=""
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815193    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815196    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815364    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815368    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815371    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815374    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815377    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815380    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815382    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815385    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815388    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815390    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815393    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815396    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815398    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:10.823860 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815400    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815403    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815405    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815408    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815410    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815413    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815415    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815418    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815421    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815423    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815426    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815428    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815432    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815435    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815438    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815440    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815443    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815446    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815448    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:10.824565 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815451    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815453    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815456    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815458    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815460    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815463    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815465    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815468    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815470    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815473    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815475    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815478    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815480    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815482    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815485    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815487    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815490    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815494    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815497    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:10.825022 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815501    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815504    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815507    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815510    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815512    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815515    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815518    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815522    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815525    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815527    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815530    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815546    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815549    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815552    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815554    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815557    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815560    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815562    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815565    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815567    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:10.825475 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815570    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815572    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815574    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815577    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815580    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815582    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815584    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815587    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815589    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815592    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815594    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815596    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815599    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815601    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815604    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815606    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815609    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815611    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815614    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:10.825955 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815618    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815621    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815623    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815626    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815629    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815633    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815637    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815640    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815643    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815645    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815648    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815650    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815653    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815655    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815658    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815660    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:10.826374 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.815686    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815724    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815728    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815731    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815733    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815736    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815738    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815741    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815743    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815746    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815749    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815751    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815753    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815756    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815758    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815761    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815764    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815767    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815769    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:10.826753 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815772    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815774    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815777    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815779    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815782    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815784    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815787    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815789    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815793    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815796    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815799    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815802    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815804    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815806    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815809    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815811    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815813    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815816    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815818    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:10.827171 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815821    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815823    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815826    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815828    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815831    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815833    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815836    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815838    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815840    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815843    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815846    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815848    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815851    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815853    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815855    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815858    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815860    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815863    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815865    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815868    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:10.827634 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815870    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815873    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815875    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815877    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815880    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815882    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815884    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815886    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815889    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815891    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815894    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815896    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815898    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815901    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815903    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815906    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815908    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815910    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815913    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815915    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:10.828081 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815917    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815920    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815922    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815925    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815927    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815929    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815932    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815934    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815937    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815939    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815943    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815947    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815951    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815954    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815956    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815959    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815961    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815964    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815966    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:10.828561 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815969    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815971    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815974    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815976    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815978    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815981    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815983    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815986    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815989    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.815992    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.824747    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.824762    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.824775    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.824780    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824834    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824839    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824842    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824846    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824851    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:10.829015 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824861    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824865    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824868    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824871    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824873    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824876    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824878    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824881    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824883    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824886    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824888    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824891    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824893    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824896    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824898    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824901    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824903    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824906    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824908    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824911    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:10.829442 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824913    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824918    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824921    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824924    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824928    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824933    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824937    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824940    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824942    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824945    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824948    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824950    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824952    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824955    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824957    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824960    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824962    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824965    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824967    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824969    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:10.829996 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824972    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824975    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824977    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824981    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824983    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824986    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824989    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824991    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824993    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824996    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.824998    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825001    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825003    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825006    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825008    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825011    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825013    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825015    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825018    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:10.830446 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825020    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825023    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825026    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825028    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825031    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825033    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825036    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825038    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825040    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825043    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825046    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825048    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825051    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825053    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825055    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825058    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825061    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825064    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825066    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825069    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:10.830922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825071    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825074    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825076    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825078    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825080    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825083    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825085    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825088    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825092    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825095    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825099    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825102    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825105    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825107    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825110    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825112    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825115    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825117    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825120    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:10.831374 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825122    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825124    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825127    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.825155    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825282    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825287    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825292    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825294    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825297    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825301    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825304    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825306    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825309    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825311    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825314    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:10.831817 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825316    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825319    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825321    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825324    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825326    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825329    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825331    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825334    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825336    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825339    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825341    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825344    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825346    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825348    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825351    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825353    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825356    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825358    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825360    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:10.832157 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825363    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825366    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825368    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825370    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825373    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825375    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825378    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825381    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825384    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825387    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825389    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825391    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825394    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825397    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825399    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825401    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825404    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825407    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825409    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825412    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:10.832596 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825414    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825416    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825419    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825421    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825424    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825426    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825428    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825431    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825433    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825435    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825438    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825440    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825442    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825445    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825447    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825450    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825452    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825454    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825457    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825459    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:10.833044 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825462    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825465    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825468    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825485    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825489    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825492    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825495    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825497    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825500    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825502    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825504    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825507    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825509    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825511    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825514    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825516    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825518    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825521    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825523    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825526    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:10.833489 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825528    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825531    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825546    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825549    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825551    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825554    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825556    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825559    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825561    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825563    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825566    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825568    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825571    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825573    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825580    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825583    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:10.834069 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.825605    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825636    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825641    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825643    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825646    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825648    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825651    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825653    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825656    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825658    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825661    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825663    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825665    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825668    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825670    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825673    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825675    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825678    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825680    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825682    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:10.834428 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825685    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825687    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825689    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825692    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825694    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825697    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825699    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825702    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825704    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825706    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825709    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825712    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825714    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825716    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825719    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825721    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825723    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825725    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825728    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825730    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:10.834922 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825732    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825735    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825737    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825739    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825742    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825744    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825746    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825749    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825751    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825753    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825756    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825758    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825761    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825763    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825765    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825768    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825770    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825772    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825774    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:10.835373 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825777    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825780    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825782    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825784    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825787    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825790    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825792    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825795    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825797    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825800    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825804    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825807    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825811    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825814    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825816    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825819    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825821    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825823    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825826    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:10.835819 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825828    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825830    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825833    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825835    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825838    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825840    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825842    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825845    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825847    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825850    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825852    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825856    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825858    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825861    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825863    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825865    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825868    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825870    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825872    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:10.836251 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825875    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825877    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825880    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825882    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825885    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825887    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825889    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825892    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825894    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:10.825896    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.825951    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.828383    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.829236    2579 server.go:1014] "Starting client certificate rotation"
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.829337    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.829379    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.831022    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:15:10.836768 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.832064    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:15:10.843593 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.843580    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:15:10.846504 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.846491    2579 log.go:25] "Validated CRI v1 image API"
Oct 30 19:15:10.847337 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.847315    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:15:10.852258 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.852237    2579 fs.go:135] Filesystem UUIDs: map[197987f7-e5df-4815-b90a-6dbfaa81f174:/dev/nvme0n1p4 2c727091-369f-4aa9-b981-5a9d4a82d4c7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:15:10.852348 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.852258    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 19:15:10.861502 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.861029    2579 manager.go:217] Machine: {Timestamp:2026-10-30 19:15:10.856372727 +0000 UTC m=+0.322281231 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3210255 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24bb9cf4f80b2cda16eff606ccc982 SystemUUID:ec24bb9c-f4f8-0b2c-da16-eff606ccc982 BootID:84af1f8f-1a5b-44e3-bb18-aa04c43351cd Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:58:ce:ce:97:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:58:ce:ce:97:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:ef:52:5a:b7:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:15:10.862093 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.862074    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:15:10.862344 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.862303    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:15:10.865358 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.865334    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:15:10.865480 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.865358    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-7.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:15:10.865553 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.865483    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:15:10.865553 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.865491    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:15:10.865553 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.865507    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:15:10.865553 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.865522    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:15:10.866202 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.866193    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:15:10.869855 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.869843    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:15:10.869965 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.869956    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:15:10.870625 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.870616    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:15:10.870653 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.870628    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:15:10.870653 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.870637    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:15:10.870653 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.870646    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:15:10.870754 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.870654    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:15:10.870966 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.870952    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:15:10.872896 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.872880    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:15:10.877176 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.877163    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:15:10.878064 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878053    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:15:10.878095 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878082    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:15:10.878227 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878218    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:15:10.878254 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878234    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:15:10.878254 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878241    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:15:10.878254 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878247    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:15:10.878254 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878254    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:15:10.878361 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878270    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:15:10.878739 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878731    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:15:10.878767 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878741    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:15:10.878767 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878749    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:15:10.878767 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878756    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:15:10.878767 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878766    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:15:10.878930 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.878776    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:15:10.879576 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.879568    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:15:10.879607 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.879579    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:15:10.883205 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.883186    2579 server.go:1263] "Started kubelet"
Oct 30 19:15:10.883302 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.883236    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:15:10.883358 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.883286    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:15:10.883403 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.883379    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:15:10.883917 ip-10-0-133-7 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:15:10.884474 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.884289    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:15:10.885586 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.885568    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:15:10.889127 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.889105    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:15:10.889449 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.889431    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:15:10.889567 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:10.889550    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-7.us-west-1.compute.internal\" not found"
Oct 30 19:15:10.890000 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.889981    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:15:10.890000 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.889984    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:15:10.890000 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.890005    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:15:10.890229 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.890215    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:15:10.890229 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.890229    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:15:10.891367 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.891349    2579 factory.go:55] Registering systemd factory
Oct 30 19:15:10.891367 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.891368    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:15:10.891518 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:10.891478    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 19:15:10.891601 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.891587    2579 factory.go:153] Registering CRI-O factory
Oct 30 19:15:10.891654 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.891603    2579 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:15:10.891654 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.891647    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:15:10.891761 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.891660    2579 factory.go:103] Registering Raw factory
Oct 30 19:15:10.891761 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.891672    2579 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:15:10.892367 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.892357    2579 manager.go:319] Starting recovery of all containers
Oct 30 19:15:10.903145 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.903121    2579 manager.go:324] Recovery completed
Oct 30 19:15:10.904178 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:10.904152    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 19:15:10.906824 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.906812    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:10.908232 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.908210    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:10.908286 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.908244    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:10.908286 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.908255    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:10.908769 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.908757    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:15:10.908769 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.908768    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:15:10.908889 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.908768    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:10.908889 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.908783    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:15:10.909434 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.909418    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:10.909490 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.909447    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:10.909490 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.909460    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:10.911041 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.911030    2579 policy_none.go:50] "Start"
Oct 30 19:15:10.911080 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.911045    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:15:10.911080 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.911055    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:15:10.913077 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.913068    2579 policy_none.go:44] "Start"
Oct 30 19:15:10.946258 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.946126    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:15:10.948727 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.948713    2579 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:15:10.948817 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:10.948746    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:15:10.948817 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.948757    2579 server.go:87] "Starting device plugin registration server"
Oct 30 19:15:10.949013 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.948999    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:15:10.949075 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.949013    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:15:10.949446 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.949211    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:15:10.949446 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.949341    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:15:10.949446 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:10.949365    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:15:10.949687 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:10.949670    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:15:10.949729 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:10.949717    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-7.us-west-1.compute.internal\" not found"
Oct 30 19:15:11.032166 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.032034    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-7.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 19:15:11.032166 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:11.032024    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-7.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 19:15:11.036312 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:11.032045    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-7.us-west-1.compute.internal.18e3649f0bbcbb36  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-7.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-7.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:15:10.88308511 +0000 UTC m=+0.348993613,LastTimestamp:2026-10-30 19:15:10.88308511 +0000 UTC m=+0.348993613,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-7.us-west-1.compute.internal,}"
Oct 30 19:15:11.057451 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.049716    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:11.057451 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.050425    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:11.057451 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.050453    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:11.057451 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.050469    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:11.057451 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.050490    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.086476 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.086425    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:15:11.087716 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.087700    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:15:11.088122 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.088107    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:15:11.088196 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.088146    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:15:11.088196 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:11.088184    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 19:15:11.099930 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.099914    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:15:11.100012 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.099962    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n4gk4"
Oct 30 19:15:11.188373 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.188342    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"]
Oct 30 19:15:11.188439 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.188410    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:11.189157 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.189145    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:11.189204 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.189171    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:11.189204 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.189186    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:11.189377 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.189359    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:11.191367 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.191355    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:11.191520 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.191506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.191585 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.191559    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:11.191634 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.191622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.191687 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.191639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.192152 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.192137    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:11.192225 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.192164    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:11.192225 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.192175    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:11.192225 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.192178    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:11.192225 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.192198    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:11.192225 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.192213    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:11.193193 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.193182    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:11.194289 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.194274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.194330 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.194305    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:11.195052 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.195038    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:11.195146 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.195071    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:11.195146 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.195092    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:11.217703 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:11.217685    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-7.us-west-1.compute.internal\" not found" node="ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.219633 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:11.219615    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-7.us-west-1.compute.internal\" not found" node="ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.290679 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.290659    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:11.291843 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.291826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.291914 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.291856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.291914 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.291886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67762efed53c0ad4c56b95d172cd37b4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"67762efed53c0ad4c56b95d172cd37b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.292006 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.291926    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.292006 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.291928    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.292701 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.292687    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.292737 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:11.292706    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-7.us-west-1.compute.internal\": node \"ip-10-0-133-7.us-west-1.compute.internal\" not found"
Oct 30 19:15:11.295223 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.294298    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:11.390636 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.390597    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.392771 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.392756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67762efed53c0ad4c56b95d172cd37b4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"67762efed53c0ad4c56b95d172cd37b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.392830 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.392801    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67762efed53c0ad4c56b95d172cd37b4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"67762efed53c0ad4c56b95d172cd37b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.451154 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.451131    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:11.451270 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.451257    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.509377 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.509358    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:11.520913 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.520893    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.523055 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.523038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:11.829247 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.829189    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 19:15:11.829755 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.829302    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:15:11.829755 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.829351    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:15:11.829755 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.829359    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 19:15:11.829755 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:11.829364    2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a99e23be2ae284760adacf985437c851-69fa068035f08793.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-7.us-west-1.compute.internal.18e3649f0d3cd8c4\": read tcp 10.0.133.7:60786->18.218.190.50:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-7.us-west-1.compute.internal.18e3649f0d3cd8c4  default   4999 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-7.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-7.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-7.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 19:15:10 +0000 UTC,LastTimestamp:2026-10-30 19:15:11.050472767 +0000 UTC m=+0.516381270,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-7.us-west-1.compute.internal,}"
Oct 30 19:15:11.871126 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.871106    2579 apiserver.go:52] "Watching apiserver"
Oct 30 19:15:11.890134 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:11.890117    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 19:15:12.104931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.104877    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:12 +0000 UTC" deadline="2028-03-28 05:49:21.000378187 +0000 UTC"
Oct 30 19:15:12.104931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.104899    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12346h34m8.895481732s"
Oct 30 19:15:12.131798 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.131776    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:15:12.132121 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.132103    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-x2pz6","openshift-multus/multus-hr8dq","openshift-network-operator/iptables-alerter-dx9vb","kube-system/konnectivity-agent-sdtv7","openshift-cluster-node-tuning-operator/tuned-n7llw","openshift-image-registry/node-ca-hlnt2","openshift-multus/multus-additional-cni-plugins-475ns","openshift-network-diagnostics/network-check-target-qk5fg","kube-system/global-pull-secret-syncer-n5gpj","openshift-multus/network-metrics-daemon-gwzbc","openshift-ovn-kubernetes/ovnkube-node-vspvz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"]
Oct 30 19:15:12.134508 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.134491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:12.136455 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.136436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.138374 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.138359    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:12.140385 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.140369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:12.142417 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.142398    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.144345 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.144330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:12.146385 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.146366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.148470 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.148453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:12.148574 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:12.148506    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:12.150567 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.150552    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:12.150644 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:12.150599    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:12.152731 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.152715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:12.152809 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:12.152778    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:12.154937 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.154920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.157191 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.157168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.186606 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.186584    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.186723 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.186619    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.186778 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.186730    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xzg6c\""
Oct 30 19:15:12.191278 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.191257    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:15:12.196921 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.196832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d18034b-20ee-4e14-9ae9-7c680ff195a1-host-slash\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:12.196921 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.196869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.196921 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.196888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-systemd-units\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.196921 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.196904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-systemd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.197133 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.196944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-log-socket\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.197133 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.196964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-bin\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.197133 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.196993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67pcp\" (UniqueName: \"kubernetes.io/projected/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-kube-api-access-67pcp\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.197133 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.197133 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cni-binary-copy\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.197133 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwkgq\" (UniqueName: \"kubernetes.io/projected/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-kube-api-access-lwkgq\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.197133 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lnrs\" (UniqueName: \"kubernetes.io/projected/9d18034b-20ee-4e14-9ae9-7c680ff195a1-kube-api-access-8lnrs\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:12.197133 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-conf\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.197133 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgzpv\" (UniqueName: \"kubernetes.io/projected/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-kube-api-access-zgzpv\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-registration-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-systemd\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-lib-modules\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv5l6\" (UniqueName: \"kubernetes.io/projected/2b725985-0e36-4850-b8ae-d761df575eb8-kube-api-access-pv5l6\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-env-overrides\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-k8s-cni-cncf-io\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197396    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-netns\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-conf-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnkqf\" (UniqueName: \"kubernetes.io/projected/30affd33-d7b4-4d40-a944-a2568dde24a0-kube-api-access-vnkqf\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-dbus\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-kubelet\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cnibin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.197626 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-serviceca\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-system-cni-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-ovn\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-netd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-kubernetes\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-host\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-tmp\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-slash\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-device-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9d18034b-20ee-4e14-9ae9-7c680ff195a1-iptables-alerter-script\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-sys\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-var-lib-kubelet\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-socket-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-sys-fs\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/866690aa-7dfb-4e2b-a29b-740a13dcc42e-konnectivity-ca\") pod \"konnectivity-agent-sdtv7\" (UID: \"866690aa-7dfb-4e2b-a29b-740a13dcc42e\") " pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:12.198179 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-var-lib-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.197991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-bin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-hostroot\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-host\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw57g\" (UniqueName: \"kubernetes.io/projected/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kube-api-access-jw57g\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-hosts-file\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-tuned\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-tmp-dir\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-system-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-socket-dir-parent\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-run\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.198819 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovn-node-metrics-cert\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198296    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-etc-kubernetes\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-etc-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/866690aa-7dfb-4e2b-a29b-740a13dcc42e-agent-certs\") pod \"konnectivity-agent-sdtv7\" (UID: \"866690aa-7dfb-4e2b-a29b-740a13dcc42e\") " pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-daemon-config\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-multus-certs\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n76vm\" (UniqueName: \"kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm\") pod \"network-check-target-qk5fg\" (UID: \"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519\") " pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-config\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-script-lib\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-multus\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysconfig\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-kubelet-config\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-node-log\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7pq5\" (UniqueName: \"kubernetes.io/projected/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-kube-api-access-j7pq5\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:12.199341 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-modprobe-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.199830 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kms8h\" (UniqueName: \"kubernetes.io/projected/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-kube-api-access-kms8h\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:12.199830 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-cnibin\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.199830 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:12.199830 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-netns\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.199830 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.199830 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.198729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-kubelet\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.287599 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.287576    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:15:12.287674 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.287608    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhxwm\""
Oct 30 19:15:12.287674 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.287631    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q7svh\""
Oct 30 19:15:12.287991 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.287979    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:15:12.288032 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.288019    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 19:15:12.288078 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.288066    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p6wlh"
Oct 30 19:15:12.288689 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.288670    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-csh58\""
Oct 30 19:15:12.288689 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.288684    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.288802 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.288779    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5vtr8\""
Oct 30 19:15:12.288802 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.288782    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-46fzl\""
Oct 30 19:15:12.288960 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.288943    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:15:12.289011 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.288961    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4k9jz\""
Oct 30 19:15:12.289011 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.288966    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:15:12.289011 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.288972    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.289112 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.289017    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6brtl\""
Oct 30 19:15:12.289112 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.289076    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:15:12.289112 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.289089    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.289373 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.289361    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:15:12.289421 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.289399    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:15:12.299718 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299696    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.299800 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299764    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:15:12.299800 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cni-binary-copy\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.299800 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299790    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.299800 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwkgq\" (UniqueName: \"kubernetes.io/projected/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-kube-api-access-lwkgq\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.299986 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lnrs\" (UniqueName: \"kubernetes.io/projected/9d18034b-20ee-4e14-9ae9-7c680ff195a1-kube-api-access-8lnrs\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:12.299986 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-conf\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.299986 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgzpv\" (UniqueName: \"kubernetes.io/projected/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-kube-api-access-zgzpv\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:12.299986 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-registration-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.299986 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-systemd\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.299986 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-lib-modules\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.299986 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299982    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.299967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5l6\" (UniqueName: \"kubernetes.io/projected/2b725985-0e36-4850-b8ae-d761df575eb8-kube-api-access-pv5l6\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300013    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-registration-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300039    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-conf\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300035    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300055    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300058    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-systemd\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-env-overrides\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-k8s-cni-cncf-io\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-netns\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300179    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-k8s-cni-cncf-io\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300196    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-netns\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-conf-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300197    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-lib-modules\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnkqf\" (UniqueName: \"kubernetes.io/projected/30affd33-d7b4-4d40-a944-a2568dde24a0-kube-api-access-vnkqf\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300205    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-conf-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.300290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-dbus\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-kubelet\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cnibin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300382    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-kubelet\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-serviceca\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300388    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-dbus\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300409    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300431    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cnibin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-system-cni-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300513    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-env-overrides\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300585    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-system-cni-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-ovn\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-netd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300636    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-ovn\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300673    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-netd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-kubernetes\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-host\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.300931 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-tmp\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300754    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-kubernetes\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300759    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-serviceca\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300766    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-host\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-slash\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-device-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300806    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-slash\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9d18034b-20ee-4e14-9ae9-7c680ff195a1-iptables-alerter-script\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300879    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-device-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300881    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-sys\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-var-lib-kubelet\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300929    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-sys\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-socket-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-sys-fs\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.300983    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-var-lib-kubelet\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.301783 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/866690aa-7dfb-4e2b-a29b-740a13dcc42e-konnectivity-ca\") pod \"konnectivity-agent-sdtv7\" (UID: \"866690aa-7dfb-4e2b-a29b-740a13dcc42e\") " pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301026    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-sys-fs\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301007    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-socket-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-var-lib-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301082    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301168    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-var-lib-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301168    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-bin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301192    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-hostroot\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:12.301259    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-host\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301272    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-bin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.302732 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301332    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-hostroot\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:12.301346    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:12.801308716 +0000 UTC m=+2.267217227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301351    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-host\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw57g\" (UniqueName: \"kubernetes.io/projected/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kube-api-access-jw57g\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-hosts-file\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301395    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9d18034b-20ee-4e14-9ae9-7c680ff195a1-iptables-alerter-script\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-tuned\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-tmp-dir\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-system-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-socket-dir-parent\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-run\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovn-node-metrics-cert\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301640    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301658    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-system-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301678    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.303473 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301729    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-socket-dir-parent\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301769    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-run\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302012    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-tmp-dir\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.301456    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-hosts-file\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-etc-kubernetes\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-etc-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/866690aa-7dfb-4e2b-a29b-740a13dcc42e-agent-certs\") pod \"konnectivity-agent-sdtv7\" (UID: \"866690aa-7dfb-4e2b-a29b-740a13dcc42e\") " pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-daemon-config\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302171    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-etc-kubernetes\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-multus-certs\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302209    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-etc-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302231    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76vm\" (UniqueName: \"kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm\") pod \"network-check-target-qk5fg\" (UID: \"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519\") " pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302250    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-multus-certs\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302281    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-config\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-script-lib\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.304255 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-multus\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysconfig\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-kubelet-config\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-node-log\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302498    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7pq5\" (UniqueName: \"kubernetes.io/projected/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-kube-api-access-j7pq5\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302625    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysconfig\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302671    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-multus\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302715    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-node-log\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-modprobe-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302761    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-kubelet-config\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kms8h\" (UniqueName: \"kubernetes.io/projected/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-kube-api-access-kms8h\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-cnibin\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302852    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302870    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-modprobe-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-netns\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.304807 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302897    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-netns\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302909    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-cnibin\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302937    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-kubelet\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d18034b-20ee-4e14-9ae9-7c680ff195a1-host-slash\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302995    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-kubelet\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.302993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-systemd-units\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-systemd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303071    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-log-socket\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303097    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d18034b-20ee-4e14-9ae9-7c680ff195a1-host-slash\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:12.302971    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-bin\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pcp\" (UniqueName: \"kubernetes.io/projected/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-kube-api-access-67pcp\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:12.303151    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret podName:d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:12.803135839 +0000 UTC m=+2.269044351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret") pod "global-pull-secret-syncer-n5gpj" (UID: "d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:12.305295 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303159    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-systemd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.305757 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303192    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-systemd-units\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.305757 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303219    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-log-socket\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.305757 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303237    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-bin\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.305757 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.305757 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303908    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-daemon-config\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.305757 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.303984    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.305757 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.304652    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-tmp\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.305757 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.304698    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-tuned\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.305757 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.304784    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/866690aa-7dfb-4e2b-a29b-740a13dcc42e-agent-certs\") pod \"konnectivity-agent-sdtv7\" (UID: \"866690aa-7dfb-4e2b-a29b-740a13dcc42e\") " pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:12.305757 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.305670    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovn-node-metrics-cert\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.310512 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.310488    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cni-binary-copy\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.336923 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.336898    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.337011 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.336949    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.337011 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.336957    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n57d7\""
Oct 30 19:15:12.337011 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.336988    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.337156 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.337014    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:15:12.337156 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.337046    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:12.337261 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.337189    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.337261 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.337254    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:15:12.337560 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.337525    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:15:12.337724 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.337710    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:15:12.342593 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.342575    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/866690aa-7dfb-4e2b-a29b-740a13dcc42e-konnectivity-ca\") pod \"konnectivity-agent-sdtv7\" (UID: \"866690aa-7dfb-4e2b-a29b-740a13dcc42e\") " pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:12.342880 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.342867    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-config\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.343016 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.343002    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-script-lib\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.359529 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.359487    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnkqf\" (UniqueName: \"kubernetes.io/projected/30affd33-d7b4-4d40-a944-a2568dde24a0-kube-api-access-vnkqf\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.359529 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.359507    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lnrs\" (UniqueName: \"kubernetes.io/projected/9d18034b-20ee-4e14-9ae9-7c680ff195a1-kube-api-access-8lnrs\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:12.360019 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.360005    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5l6\" (UniqueName: \"kubernetes.io/projected/2b725985-0e36-4850-b8ae-d761df575eb8-kube-api-access-pv5l6\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.360056 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.360006    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgzpv\" (UniqueName: \"kubernetes.io/projected/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-kube-api-access-zgzpv\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:12.362394 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:12.362362    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:12.362472 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:12.362398    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:12.362472 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:12.362411    2579 projected.go:196] Error preparing data for projected volume kube-api-access-n76vm for pod openshift-network-diagnostics/network-check-target-qk5fg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:12.362562 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:12.362490    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm podName:c4b32629-b4f3-4bb5-b8f2-ef5d0b634519 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:12.862473023 +0000 UTC m=+2.328381533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n76vm" (UniqueName: "kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm") pod "network-check-target-qk5fg" (UID: "c4b32629-b4f3-4bb5-b8f2-ef5d0b634519") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:12.363644 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.363621    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwkgq\" (UniqueName: \"kubernetes.io/projected/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-kube-api-access-lwkgq\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.364144 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.364126    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7pq5\" (UniqueName: \"kubernetes.io/projected/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-kube-api-access-j7pq5\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:12.364904 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.364885    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pcp\" (UniqueName: \"kubernetes.io/projected/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-kube-api-access-67pcp\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.365919 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.365904    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw57g\" (UniqueName: \"kubernetes.io/projected/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kube-api-access-jw57g\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.368453 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.368428    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kms8h\" (UniqueName: \"kubernetes.io/projected/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-kube-api-access-kms8h\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:12.462793 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.462770    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:12.476757 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.476736    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:12.494337 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.494306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:12.502069 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.502051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:12.521260 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.521223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:12.542741 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.542717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:12.570776 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.570748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:12.596503 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.596486    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:12.598668 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.598652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:12.602493 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:12.602465    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d302d841e966ec6a147c94830317a7.slice/crio-af3b653c88e7f39f1cc9e214dbeccfe70cef031b0bdf5d25678caa77179c4065 WatchSource:0}: Error finding container af3b653c88e7f39f1cc9e214dbeccfe70cef031b0bdf5d25678caa77179c4065: Status 404 returned error can't find the container with id af3b653c88e7f39f1cc9e214dbeccfe70cef031b0bdf5d25678caa77179c4065
Oct 30 19:15:12.603043 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:12.603022    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67762efed53c0ad4c56b95d172cd37b4.slice/crio-d128c72620004e303aa56724f53e24f277d58fecc5934d7d52538135626319de WatchSource:0}: Error finding container d128c72620004e303aa56724f53e24f277d58fecc5934d7d52538135626319de: Status 404 returned error can't find the container with id d128c72620004e303aa56724f53e24f277d58fecc5934d7d52538135626319de
Oct 30 19:15:12.603760 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:12.603741    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b725985_0e36_4850_b8ae_d761df575eb8.slice/crio-22d296ffb312dfb38f65b32209805e059e6f47fcf36503b50fbfa6e6a809c8cb WatchSource:0}: Error finding container 22d296ffb312dfb38f65b32209805e059e6f47fcf36503b50fbfa6e6a809c8cb: Status 404 returned error can't find the container with id 22d296ffb312dfb38f65b32209805e059e6f47fcf36503b50fbfa6e6a809c8cb
Oct 30 19:15:12.608594 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:12.608578    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:15:12.610840 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:12.610805    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a8605a_fd55_4808_b74c_cd77b35fdbd3.slice/crio-a3a8254b28241ccbd5d240415df752f9cebe1cf720577e448c8d896820bf0ab5 WatchSource:0}: Error finding container a3a8254b28241ccbd5d240415df752f9cebe1cf720577e448c8d896820bf0ab5: Status 404 returned error can't find the container with id a3a8254b28241ccbd5d240415df752f9cebe1cf720577e448c8d896820bf0ab5
Oct 30 19:15:12.611324 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:12.611306    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bd44c9_6ec3_4eaf_82fc_0d8f7c1264be.slice/crio-2131acb0cb45e134376f22dc901d68cffaa7848b03f680182309b94cd65ddb74 WatchSource:0}: Error finding container 2131acb0cb45e134376f22dc901d68cffaa7848b03f680182309b94cd65ddb74: Status 404 returned error can't find the container with id 2131acb0cb45e134376f22dc901d68cffaa7848b03f680182309b94cd65ddb74
Oct 30 19:15:14.731202 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:14.731178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:14.731290 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:14.731224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:14.731366 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:14.731298    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:14.731366 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:14.731320    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:14.731366 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:14.731351    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:15.731337431 +0000 UTC m=+3.271968147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:14.731458 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:14.731367    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret podName:d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:15.731358564 +0000 UTC m=+3.271989267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret") pod "global-pull-secret-syncer-n5gpj" (UID: "d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:14.831728 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:14.831677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76vm\" (UniqueName: \"kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm\") pod \"network-check-target-qk5fg\" (UID: \"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519\") " pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:14.832163 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:14.831778    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:14.832163 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:14.831790    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:14.832163 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:14.831798    2579 projected.go:196] Error preparing data for projected volume kube-api-access-n76vm for pod openshift-network-diagnostics/network-check-target-qk5fg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:14.832163 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:14.831853    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm podName:c4b32629-b4f3-4bb5-b8f2-ef5d0b634519 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:15.831842721 +0000 UTC m=+3.372473434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n76vm" (UniqueName: "kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm") pod "network-check-target-qk5fg" (UID: "c4b32629-b4f3-4bb5-b8f2-ef5d0b634519") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:15.017003 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.016960    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-475ns" event={"ID":"2b725985-0e36-4850-b8ae-d761df575eb8","Type":"ContainerStarted","Data":"22d296ffb312dfb38f65b32209805e059e6f47fcf36503b50fbfa6e6a809c8cb"}
Oct 30 19:15:15.017986 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.017965    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal" event={"ID":"95d302d841e966ec6a147c94830317a7","Type":"ContainerStarted","Data":"af3b653c88e7f39f1cc9e214dbeccfe70cef031b0bdf5d25678caa77179c4065"}
Oct 30 19:15:15.019021 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.019001    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz" event={"ID":"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be","Type":"ContainerStarted","Data":"2131acb0cb45e134376f22dc901d68cffaa7848b03f680182309b94cd65ddb74"}
Oct 30 19:15:15.019979 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.019963    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8" event={"ID":"98a8605a-fd55-4808-b74c-cd77b35fdbd3","Type":"ContainerStarted","Data":"a3a8254b28241ccbd5d240415df752f9cebe1cf720577e448c8d896820bf0ab5"}
Oct 30 19:15:15.020878 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.020861    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal" event={"ID":"67762efed53c0ad4c56b95d172cd37b4","Type":"ContainerStarted","Data":"d128c72620004e303aa56724f53e24f277d58fecc5934d7d52538135626319de"}
Oct 30 19:15:15.153064 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.153010    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:15.185534 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:15.185509    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d3697f_f464_4a7d_af17_4ea6d73ae5ac.slice/crio-a8040d9deb7b2993592593928dcb5f1a0ee9a9c499dfbcd7f198ab9689add99c WatchSource:0}: Error finding container a8040d9deb7b2993592593928dcb5f1a0ee9a9c499dfbcd7f198ab9689add99c: Status 404 returned error can't find the container with id a8040d9deb7b2993592593928dcb5f1a0ee9a9c499dfbcd7f198ab9689add99c
Oct 30 19:15:15.209248 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:15.209232    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d18034b_20ee_4e14_9ae9_7c680ff195a1.slice/crio-a587ed1ea2bd536434d626e3121020adafb9ba0a664580fe92d66ae4d1d58fa6 WatchSource:0}: Error finding container a587ed1ea2bd536434d626e3121020adafb9ba0a664580fe92d66ae4d1d58fa6: Status 404 returned error can't find the container with id a587ed1ea2bd536434d626e3121020adafb9ba0a664580fe92d66ae4d1d58fa6
Oct 30 19:15:15.216489 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.216452    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:13 +0000 UTC" deadline="2028-05-31 09:06:24.643656133 +0000 UTC"
Oct 30 19:15:15.216489 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.216488    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13885h51m9.427170627s"
Oct 30 19:15:15.278875 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:15.278851    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c29e8f_e445_40b1_a2c0_1e8252b8f53f.slice/crio-8548a5f242ca23929b1fae7d7b633ab559c8d8332fc958cb61d548d44b322ba2 WatchSource:0}: Error finding container 8548a5f242ca23929b1fae7d7b633ab559c8d8332fc958cb61d548d44b322ba2: Status 404 returned error can't find the container with id 8548a5f242ca23929b1fae7d7b633ab559c8d8332fc958cb61d548d44b322ba2
Oct 30 19:15:15.296568 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.296551    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:15.386608 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:15.386569    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093a7b2a_fb5b_4703_8073_98c60f1ef5ca.slice/crio-7a707cf7c9283c7172f415b55512868f664462a44150b95aaef1afdf6bbad56f WatchSource:0}: Error finding container 7a707cf7c9283c7172f415b55512868f664462a44150b95aaef1afdf6bbad56f: Status 404 returned error can't find the container with id 7a707cf7c9283c7172f415b55512868f664462a44150b95aaef1afdf6bbad56f
Oct 30 19:15:15.388131 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.388108    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:15.739251 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.739213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:15.739433 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.739282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:15.739433 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:15.739400    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:15.739545 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:15.739457    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret podName:d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:17.739441051 +0000 UTC m=+5.280071767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret") pod "global-pull-secret-syncer-n5gpj" (UID: "d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:15.739545 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:15.739522    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:15.739656 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:15.739560    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:17.739548284 +0000 UTC m=+5.280178997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:15.839895 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:15.839799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76vm\" (UniqueName: \"kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm\") pod \"network-check-target-qk5fg\" (UID: \"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519\") " pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:15.840457 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:15.839959    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:15.840457 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:15.839988    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:15.840457 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:15.840001    2579 projected.go:196] Error preparing data for projected volume kube-api-access-n76vm for pod openshift-network-diagnostics/network-check-target-qk5fg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:15.840457 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:15.840062    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm podName:c4b32629-b4f3-4bb5-b8f2-ef5d0b634519 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:17.840043002 +0000 UTC m=+5.380673720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n76vm" (UniqueName: "kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm") pod "network-check-target-qk5fg" (UID: "c4b32629-b4f3-4bb5-b8f2-ef5d0b634519") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:16.014714 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:16.014231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:16.014714 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:16.014367    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:16.014714 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:16.014439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:16.014714 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:16.014505    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:16.014714 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:16.014558    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:16.014714 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:16.014632    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:16.034230 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:16.034195    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlnt2" event={"ID":"093a7b2a-fb5b-4703-8073-98c60f1ef5ca","Type":"ContainerStarted","Data":"7a707cf7c9283c7172f415b55512868f664462a44150b95aaef1afdf6bbad56f"}
Oct 30 19:15:16.038287 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:16.038247    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hr8dq" event={"ID":"75c29e8f-e445-40b1-a2c0-1e8252b8f53f","Type":"ContainerStarted","Data":"8548a5f242ca23929b1fae7d7b633ab559c8d8332fc958cb61d548d44b322ba2"}
Oct 30 19:15:16.039630 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:16.039607    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sdtv7" event={"ID":"866690aa-7dfb-4e2b-a29b-740a13dcc42e","Type":"ContainerStarted","Data":"ed2647f3f8def4c27969647f9054891824477ecaa703c6d0bae0cc9c7cd362b3"}
Oct 30 19:15:16.042140 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:16.042112    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dx9vb" event={"ID":"9d18034b-20ee-4e14-9ae9-7c680ff195a1","Type":"ContainerStarted","Data":"a587ed1ea2bd536434d626e3121020adafb9ba0a664580fe92d66ae4d1d58fa6"}
Oct 30 19:15:16.043468 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:16.043447    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x2pz6" event={"ID":"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac","Type":"ContainerStarted","Data":"a8040d9deb7b2993592593928dcb5f1a0ee9a9c499dfbcd7f198ab9689add99c"}
Oct 30 19:15:16.045504 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:16.045482    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n7llw" event={"ID":"30affd33-d7b4-4d40-a944-a2568dde24a0","Type":"ContainerStarted","Data":"5ed5e4fba348b38de371d1430df3b522bed098c0806b94ecc897a3fa53472e91"}
Oct 30 19:15:16.217102 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:16.217031    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:13 +0000 UTC" deadline="2028-04-13 11:56:35.62615574 +0000 UTC"
Oct 30 19:15:16.217102 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:16.217085    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12736h41m19.409074396s"
Oct 30 19:15:17.754829 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:17.754523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:17.754829 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:17.754776    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:17.754829 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:17.754805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:17.755355 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:17.754865    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:21.754846452 +0000 UTC m=+9.295477167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:17.755355 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:17.754918    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:17.755355 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:17.754968    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret podName:d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:21.754953828 +0000 UTC m=+9.295584550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret") pod "global-pull-secret-syncer-n5gpj" (UID: "d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:17.856244 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:17.856213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76vm\" (UniqueName: \"kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm\") pod \"network-check-target-qk5fg\" (UID: \"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519\") " pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:17.856447 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:17.856404    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:17.856447 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:17.856433    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:17.856447 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:17.856444    2579 projected.go:196] Error preparing data for projected volume kube-api-access-n76vm for pod openshift-network-diagnostics/network-check-target-qk5fg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:17.856620 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:17.856496    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm podName:c4b32629-b4f3-4bb5-b8f2-ef5d0b634519 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:21.856480161 +0000 UTC m=+9.397110878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n76vm" (UniqueName: "kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm") pod "network-check-target-qk5fg" (UID: "c4b32629-b4f3-4bb5-b8f2-ef5d0b634519") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:18.014548 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:18.014476    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:18.014663 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:18.014591    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:18.014663 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:18.014647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:18.014726 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:18.014687    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:18.014763 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:18.014733    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:18.014792 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:18.014771    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:18.060521 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:18.060465    2579 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="7083dd6cc684392f5c4d01ec21a92eec95baaa44b62b0d8efb416bb8dbcdea9f" exitCode=0
Oct 30 19:15:18.060679 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:18.060526    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-475ns" event={"ID":"2b725985-0e36-4850-b8ae-d761df575eb8","Type":"ContainerDied","Data":"7083dd6cc684392f5c4d01ec21a92eec95baaa44b62b0d8efb416bb8dbcdea9f"}
Oct 30 19:15:19.076926 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:19.076130    2579 generic.go:356] "Generic (PLEG): container finished" podID="95d302d841e966ec6a147c94830317a7" containerID="3f0830ec228c8c1422f267d8e79c4542e529fbcfe83318f742ccce8169b89c08" exitCode=0
Oct 30 19:15:19.076926 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:19.076220    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal" event={"ID":"95d302d841e966ec6a147c94830317a7","Type":"ContainerDied","Data":"3f0830ec228c8c1422f267d8e79c4542e529fbcfe83318f742ccce8169b89c08"}
Oct 30 19:15:19.082863 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:19.082648    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal" event={"ID":"67762efed53c0ad4c56b95d172cd37b4","Type":"ContainerStarted","Data":"e4ffb23a4a050f7a6a61b2c16f7769c73038394c5bb0da3abb9367a6d263f734"}
Oct 30 19:15:20.014455 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:20.014397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:20.014657 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:20.014538    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:20.014657 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:20.014598    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:20.014733 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:20.014669    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:20.014733 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:20.014711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:20.014796 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:20.014781    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:20.103116 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:20.103081    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sdtv7" event={"ID":"866690aa-7dfb-4e2b-a29b-740a13dcc42e","Type":"ContainerStarted","Data":"ac5b130b01616526531e51d975c0c3eb5bbbe4604769709cb862db1425d1b6da"}
Oct 30 19:15:20.119027 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:20.118996    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal" event={"ID":"95d302d841e966ec6a147c94830317a7","Type":"ContainerStarted","Data":"e88e54d6b3d66d91754a3d9294bc73a3029b25ae773520efbb2af234c90c52e7"}
Oct 30 19:15:20.216066 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:20.215999    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal" podStartSLOduration=7.215979926 podStartE2EDuration="7.215979926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:19.301900051 +0000 UTC m=+6.842530774" watchObservedRunningTime="2026-10-30 19:15:20.215979926 +0000 UTC m=+7.756610650"
Oct 30 19:15:20.330280 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:20.330180    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sdtv7" podStartSLOduration=3.143298054 podStartE2EDuration="7.330157901s" totalImagesPullingTime="4.186859847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:20.216934715 +0000 UTC m=+7.757565438" watchObservedRunningTime="2026-10-30 19:15:20.330157901 +0000 UTC m=+7.870788624"
Oct 30 19:15:21.125874 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:21.125367    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8" event={"ID":"98a8605a-fd55-4808-b74c-cd77b35fdbd3","Type":"ContainerStarted","Data":"c24c86b565b7da2ee1a976cb6834dd1d3aff9286056ad5f9850fec06dbc45aa4"}
Oct 30 19:15:21.451296 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:21.450914    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:21.451447 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:21.451427    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:21.564684 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:21.564627    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal" podStartSLOduration=8.5646074 podStartE2EDuration="8.5646074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:20.333498489 +0000 UTC m=+7.874129212" watchObservedRunningTime="2026-10-30 19:15:21.5646074 +0000 UTC m=+9.105238123"
Oct 30 19:15:21.792788 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:21.792705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:21.792788 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:21.792782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:21.793022 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:21.792942    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:21.793022 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:21.793003    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret podName:d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:29.792986356 +0000 UTC m=+17.333617077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret") pod "global-pull-secret-syncer-n5gpj" (UID: "d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:21.793444 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:21.793423    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:21.793559 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:21.793474    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:29.793460202 +0000 UTC m=+17.334090917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:21.893846 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:21.893795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76vm\" (UniqueName: \"kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm\") pod \"network-check-target-qk5fg\" (UID: \"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519\") " pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:21.894019 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:21.893978    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:21.894019 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:21.893997    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:21.894019 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:21.894008    2579 projected.go:196] Error preparing data for projected volume kube-api-access-n76vm for pod openshift-network-diagnostics/network-check-target-qk5fg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:21.894194 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:21.894066    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm podName:c4b32629-b4f3-4bb5-b8f2-ef5d0b634519 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:29.894048232 +0000 UTC m=+17.434678947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n76vm" (UniqueName: "kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm") pod "network-check-target-qk5fg" (UID: "c4b32629-b4f3-4bb5-b8f2-ef5d0b634519") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:22.014357 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:22.014322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:22.014526 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:22.014460    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:22.014926 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:22.014872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:22.015049 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:22.014971    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:22.015049 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:22.015039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:22.015163 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:22.015109    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:22.130714 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:22.130644    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlnt2" event={"ID":"093a7b2a-fb5b-4703-8073-98c60f1ef5ca","Type":"ContainerStarted","Data":"3dde4b7c65f6c5d8b299db650ff77736d283ec1a84cf9131bf6d5878c0f261e7"}
Oct 30 19:15:22.694508 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:22.694485    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:15:22.879621 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:22.879514    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:15:22.694505584Z","UUID":"973aa20e-da35-4ba2-a956-92dcbbb06ea7","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:15:22.881232 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:22.881210    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:15:22.881343 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:22.881241    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:15:23.133981 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:23.133932    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8" event={"ID":"98a8605a-fd55-4808-b74c-cd77b35fdbd3","Type":"ContainerStarted","Data":"44f139fc0819d9af4765193e18d361556c2924c9956adcafd06e2383b0f5370b"}
Oct 30 19:15:23.138579 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:23.138560    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:15:23.139579 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:23.139554    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-475ns" event={"ID":"2b725985-0e36-4850-b8ae-d761df575eb8","Type":"ContainerStarted","Data":"04d64e64975c16ea629d1f9847ed4cf1768f60e9fe788643ab101e9370f35815"}
Oct 30 19:15:23.258372 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:23.258316    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hlnt2" podStartSLOduration=4.623758013 podStartE2EDuration="10.258297516s" totalImagesPullingTime="5.634539503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:22.242242833 +0000 UTC m=+9.782873559" watchObservedRunningTime="2026-10-30 19:15:23.258297516 +0000 UTC m=+10.798928240"
Oct 30 19:15:24.014411 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:24.014213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:24.014581 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:24.014238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:24.014581 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:24.014523    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:24.014702 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:24.014594    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:24.014702 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:24.014253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:24.014702 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:24.014691    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:24.141785 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:24.141746    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hr8dq" event={"ID":"75c29e8f-e445-40b1-a2c0-1e8252b8f53f","Type":"ContainerStarted","Data":"6ccd3d4b9be2d7c3c18eeff4cf16a8cc7607ec0163e84328943433120a9111ac"}
Oct 30 19:15:24.143375 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:24.143346    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x2pz6" event={"ID":"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac","Type":"ContainerStarted","Data":"fd629f4c6c7b6215f2207f9f75f9153557575b17e575f981d55a0f22d9f7b3a5"}
Oct 30 19:15:24.145315 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:24.145289    2579 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="04d64e64975c16ea629d1f9847ed4cf1768f60e9fe788643ab101e9370f35815" exitCode=0
Oct 30 19:15:24.145415 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:24.145322    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-475ns" event={"ID":"2b725985-0e36-4850-b8ae-d761df575eb8","Type":"ContainerDied","Data":"04d64e64975c16ea629d1f9847ed4cf1768f60e9fe788643ab101e9370f35815"}
Oct 30 19:15:24.260573 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:24.260412    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hr8dq" podStartSLOduration=3.202659516 podStartE2EDuration="11.260400216s" totalImagesPullingTime="8.0577407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:24.259525727 +0000 UTC m=+11.800156449" watchObservedRunningTime="2026-10-30 19:15:24.260400216 +0000 UTC m=+11.801030937"
Oct 30 19:15:24.377306 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:24.377235    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-x2pz6" podStartSLOduration=3.369504957 podStartE2EDuration="11.377220373s" totalImagesPullingTime="8.007715416s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:24.377087613 +0000 UTC m=+11.917718335" watchObservedRunningTime="2026-10-30 19:15:24.377220373 +0000 UTC m=+11.917851096"
Oct 30 19:15:25.150011 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:25.149935    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dx9vb" event={"ID":"9d18034b-20ee-4e14-9ae9-7c680ff195a1","Type":"ContainerStarted","Data":"5156cefeb09b4a20bcafd82f2cbbbd6f8961031139f6324b73cabefa393a3de7"}
Oct 30 19:15:25.152719 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:25.152681    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8" event={"ID":"98a8605a-fd55-4808-b74c-cd77b35fdbd3","Type":"ContainerStarted","Data":"060f3eda1759da3bf6a439bba0074a503562775ee5c3e3aa9085cdd87afb2202"}
Oct 30 19:15:25.375165 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:25.375108    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8" podStartSLOduration=3.184164301 podStartE2EDuration="12.375089567s" totalImagesPullingTime="9.190925266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:25.375051449 +0000 UTC m=+12.915682228" watchObservedRunningTime="2026-10-30 19:15:25.375089567 +0000 UTC m=+12.915720289"
Oct 30 19:15:25.375416 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:25.375391    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dx9vb" podStartSLOduration=4.286791076 podStartE2EDuration="12.375379934s" totalImagesPullingTime="8.088588858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:25.261691835 +0000 UTC m=+12.802322556" watchObservedRunningTime="2026-10-30 19:15:25.375379934 +0000 UTC m=+12.916010657"
Oct 30 19:15:26.013912 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:26.013879    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:26.013912 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:26.013902    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:26.014128 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:26.013894    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:26.014128 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:26.014011    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:26.014235 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:26.014168    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:26.014325 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:26.014297    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:26.156824 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:26.156793    2579 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="5f8e9bafd05ee42b7169a28a96a8f9be885c79ceb673f1a913aba2bf8bfd9b60" exitCode=0
Oct 30 19:15:26.157522 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:26.156894    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-475ns" event={"ID":"2b725985-0e36-4850-b8ae-d761df575eb8","Type":"ContainerDied","Data":"5f8e9bafd05ee42b7169a28a96a8f9be885c79ceb673f1a913aba2bf8bfd9b60"}
Oct 30 19:15:26.158885 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:26.158831    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n7llw" event={"ID":"30affd33-d7b4-4d40-a944-a2568dde24a0","Type":"ContainerStarted","Data":"7e02b3e20a02e80462e0b1f418647d179b0f274c685f3eef2f467a89f12ab0ec"}
Oct 30 19:15:26.487399 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:26.487311    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n7llw" podStartSLOduration=3.427384763 podStartE2EDuration="13.487296277s" totalImagesPullingTime="10.059911514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:26.487030942 +0000 UTC m=+14.027661664" watchObservedRunningTime="2026-10-30 19:15:26.487296277 +0000 UTC m=+14.027926999"
Oct 30 19:15:28.014144 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:28.013875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:28.014678 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:28.013875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:28.014678 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:28.014160    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:28.014678 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:28.014253    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:28.014678 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:28.013923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:28.014678 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:28.014346    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:28.166667 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:28.166624    2579 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="04799b3026c89376f49d3236c697d47ca00ae4783e87e00fe1b3f3e31afb29d0" exitCode=0
Oct 30 19:15:28.166903 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:28.166674    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-475ns" event={"ID":"2b725985-0e36-4850-b8ae-d761df575eb8","Type":"ContainerDied","Data":"04799b3026c89376f49d3236c697d47ca00ae4783e87e00fe1b3f3e31afb29d0"}
Oct 30 19:15:29.866113 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:29.866071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:29.866560 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:29.866128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:29.866560 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:29.866215    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:29.866560 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:29.866276    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.866261354 +0000 UTC m=+33.406892058 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 19:15:29.866560 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:29.866215    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:29.866560 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:29.866343    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret podName:d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.866331677 +0000 UTC m=+33.406962381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret") pod "global-pull-secret-syncer-n5gpj" (UID: "d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51") : object "kube-system"/"original-pull-secret" not registered
Oct 30 19:15:29.966437 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:29.966395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76vm\" (UniqueName: \"kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm\") pod \"network-check-target-qk5fg\" (UID: \"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519\") " pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:29.966587 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:29.966533    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 19:15:29.966587 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:29.966548    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 19:15:29.966587 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:29.966569    2579 projected.go:196] Error preparing data for projected volume kube-api-access-n76vm for pod openshift-network-diagnostics/network-check-target-qk5fg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:29.966695 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:29.966629    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm podName:c4b32629-b4f3-4bb5-b8f2-ef5d0b634519 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.966613996 +0000 UTC m=+33.507244712 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n76vm" (UniqueName: "kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm") pod "network-check-target-qk5fg" (UID: "c4b32629-b4f3-4bb5-b8f2-ef5d0b634519") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 19:15:30.014371 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:30.014344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:30.014539 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:30.014374    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:30.014539 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:30.014348    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:30.014539 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:30.014486    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:30.014688 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:30.014604    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:30.014688 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:30.014675    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:31.175507 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:31.175311    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz" event={"ID":"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be","Type":"ContainerStarted","Data":"bff8684363abc2f9b1e6285f26a342c2a43561b07e6b0f0cae6bf594d9347fa1"}
Oct 30 19:15:31.175924 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:31.175557    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz" event={"ID":"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be","Type":"ContainerStarted","Data":"6b36539137f32da1f5c88ee9e8522cf3b9f1ee660234ff4c9a4c4d7bde1abe0c"}
Oct 30 19:15:31.175924 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:31.175591    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz" event={"ID":"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be","Type":"ContainerStarted","Data":"277317b467c862faa1ddbf1924804fc9544111f304f54cd66b4c9fd101e33104"}
Oct 30 19:15:31.175924 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:31.175629    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz" event={"ID":"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be","Type":"ContainerStarted","Data":"c12b8acec069c65e21e91d74844a00db1f6e7b8f48e1eedcddcd17ceddf02687"}
Oct 30 19:15:31.175924 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:31.175655    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz" event={"ID":"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be","Type":"ContainerStarted","Data":"362b123d1b8ecc21e57bbdb9283f6d3ae6d2b0f5adf8c81599e56fb8971e30b0"}
Oct 30 19:15:31.175924 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:31.175689    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz" event={"ID":"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be","Type":"ContainerStarted","Data":"cb41311c54b16920e5e83c17502991e5587a26f387766d4cc91d0bcff262a8dd"}
Oct 30 19:15:31.178022 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:31.177991    2579 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="1fa259238fc7bfed4e5b04995e44bd17f2f3ed47e901fcaf7349e099409901c5" exitCode=0
Oct 30 19:15:31.178132 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:31.178025    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-475ns" event={"ID":"2b725985-0e36-4850-b8ae-d761df575eb8","Type":"ContainerDied","Data":"1fa259238fc7bfed4e5b04995e44bd17f2f3ed47e901fcaf7349e099409901c5"}
Oct 30 19:15:32.013616 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:32.013585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:32.013616 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:32.013610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:32.013802 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:32.013698    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:32.013802 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:32.013702    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:32.013802 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:32.013760    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:32.013905 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:32.013837    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:32.182714 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:32.182685    2579 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="a8c4ce81a2d4d47e9100a03473e7258ec653aa6173bb4b9203c22197a735f8b4" exitCode=0
Oct 30 19:15:32.183044 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:32.182733    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-475ns" event={"ID":"2b725985-0e36-4850-b8ae-d761df575eb8","Type":"ContainerDied","Data":"a8c4ce81a2d4d47e9100a03473e7258ec653aa6173bb4b9203c22197a735f8b4"}
Oct 30 19:15:33.187468 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:33.187433    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz" event={"ID":"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be","Type":"ContainerStarted","Data":"0846f57bfc670d516f99dd4dd12ad7bac851aafe1e26316d7811706df45c5a3c"}
Oct 30 19:15:33.190343 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:33.190322    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-475ns" event={"ID":"2b725985-0e36-4850-b8ae-d761df575eb8","Type":"ContainerStarted","Data":"529d92e9c26e372bd3e40f7defa70a5f3ab99f99dbe41f8760c980debcff3765"}
Oct 30 19:15:33.308021 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:33.307979    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-475ns" podStartSLOduration=8.614214761 podStartE2EDuration="20.307967309s" totalImagesPullingTime="11.693752548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:33.307743098 +0000 UTC m=+20.848373821" watchObservedRunningTime="2026-10-30 19:15:33.307967309 +0000 UTC m=+20.848598024"
Oct 30 19:15:34.014297 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:34.014274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:34.014440 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:34.014312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:34.014440 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:34.014393    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:34.014530 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:34.014438    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:34.014530 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:34.014483    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:34.014601 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:34.014570    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:34.501032 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:34.501010    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:34.501394 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:34.501104    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:15:34.501495 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:34.501482    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:35.196877 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:35.196845    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz" event={"ID":"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be","Type":"ContainerStarted","Data":"f7f3677515ca8ff50bfe4decaeb1d48199016ee2e2ea9ea9338cec708a7402e9"}
Oct 30 19:15:35.197263 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:35.197239    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:35.197393 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:35.197319    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:35.212366 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:35.212343    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:35.470665 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:35.470581    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz" podStartSLOduration=6.657815864 podStartE2EDuration="22.470568847s" totalImagesPullingTime="15.812752983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:35.470080524 +0000 UTC m=+23.010711246" watchObservedRunningTime="2026-10-30 19:15:35.470568847 +0000 UTC m=+23.011199569"
Oct 30 19:15:36.013744 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:36.013722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:36.014362 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:36.013797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:36.014362 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:36.013891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:36.014362 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:36.013910    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:36.014362 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:36.013951    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:36.014362 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:36.014005    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:36.199720 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:36.199684    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:36.214452 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:36.214431    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:37.685699 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:37.685454    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gwzbc"]
Oct 30 19:15:37.686123 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:37.685926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:37.686263 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:37.686237    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:37.734737 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:37.734711    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qk5fg"]
Oct 30 19:15:37.734864 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:37.734802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:37.734919 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:37.734889    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:37.735265 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:37.735248    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n5gpj"]
Oct 30 19:15:37.735335 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:37.735325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:37.735404 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:37.735389    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:39.014249 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:39.014219    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:39.014249 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:39.014239    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:39.014664 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:39.014219    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:39.014664 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:39.014354    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:39.014664 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:39.014446    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:39.014664 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:39.014508    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:41.014173 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:41.014146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:41.014173 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:41.014159    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:41.014618 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:41.014146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:41.014618 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:41.014237    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:41.014618 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:41.014325    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:41.014618 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:41.014402    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:43.015243 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.015216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:43.015662 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:43.015296    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qk5fg" podUID="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519"
Oct 30 19:15:43.015662 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.015323    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:43.015662 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.015393    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:43.015662 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:43.015481    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwzbc" podUID="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f"
Oct 30 19:15:43.015662 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:43.015521    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n5gpj" podUID="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51"
Oct 30 19:15:43.175231 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.175168    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeReady"
Oct 30 19:15:43.175344 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.175287    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 19:15:43.329795 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.329767    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2gdtt"]
Oct 30 19:15:43.334195 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.334175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.334877 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.334859    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d9llh"]
Oct 30 19:15:43.338169 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.338147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:43.392909 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.392892    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:15:43.393167 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.393144    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:15:43.393265 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.393172    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:15:43.393265 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.393196    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qjhd5\""
Oct 30 19:15:43.393346 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.393283    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hk6fn\""
Oct 30 19:15:43.396875 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.396862    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:15:43.397716 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.397700    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2gdtt"]
Oct 30 19:15:43.404072 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.403925    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d9llh"]
Oct 30 19:15:43.461766 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.461716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb9780a-2710-4742-b890-b7c0647a4086-tmp-dir\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.461766 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.461762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd392127-738d-43fd-8e0a-f6e207feaf4a-cert\") pod \"ingress-canary-d9llh\" (UID: \"fd392127-738d-43fd-8e0a-f6e207feaf4a\") " pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:43.461915 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.461864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb9780a-2710-4742-b890-b7c0647a4086-config-volume\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.461915 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.461894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgz84\" (UniqueName: \"kubernetes.io/projected/fd392127-738d-43fd-8e0a-f6e207feaf4a-kube-api-access-wgz84\") pod \"ingress-canary-d9llh\" (UID: \"fd392127-738d-43fd-8e0a-f6e207feaf4a\") " pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:43.461996 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.461918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.461996 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.461972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4zj\" (UniqueName: \"kubernetes.io/projected/ffb9780a-2710-4742-b890-b7c0647a4086-kube-api-access-vn4zj\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.489144 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.489122    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:15:43.563209 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.563189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.563316 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.563223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4zj\" (UniqueName: \"kubernetes.io/projected/ffb9780a-2710-4742-b890-b7c0647a4086-kube-api-access-vn4zj\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.563316 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.563251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb9780a-2710-4742-b890-b7c0647a4086-tmp-dir\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.563316 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.563276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd392127-738d-43fd-8e0a-f6e207feaf4a-cert\") pod \"ingress-canary-d9llh\" (UID: \"fd392127-738d-43fd-8e0a-f6e207feaf4a\") " pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:43.563465 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:43.563328    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:43.563465 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:43.563382    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls podName:ffb9780a-2710-4742-b890-b7c0647a4086 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:44.06336286 +0000 UTC m=+31.603993571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls") pod "dns-default-2gdtt" (UID: "ffb9780a-2710-4742-b890-b7c0647a4086") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:43.563465 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.563411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb9780a-2710-4742-b890-b7c0647a4086-config-volume\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.563465 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.563445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgz84\" (UniqueName: \"kubernetes.io/projected/fd392127-738d-43fd-8e0a-f6e207feaf4a-kube-api-access-wgz84\") pod \"ingress-canary-d9llh\" (UID: \"fd392127-738d-43fd-8e0a-f6e207feaf4a\") " pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:43.563629 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.563532    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb9780a-2710-4742-b890-b7c0647a4086-tmp-dir\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.563935 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.563919    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb9780a-2710-4742-b890-b7c0647a4086-config-volume\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.566989 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.566969    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd392127-738d-43fd-8e0a-f6e207feaf4a-cert\") pod \"ingress-canary-d9llh\" (UID: \"fd392127-738d-43fd-8e0a-f6e207feaf4a\") " pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:43.621077 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.621055    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4zj\" (UniqueName: \"kubernetes.io/projected/ffb9780a-2710-4742-b890-b7c0647a4086-kube-api-access-vn4zj\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:43.622766 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.622746    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgz84\" (UniqueName: \"kubernetes.io/projected/fd392127-738d-43fd-8e0a-f6e207feaf4a-kube-api-access-wgz84\") pod \"ingress-canary-d9llh\" (UID: \"fd392127-738d-43fd-8e0a-f6e207feaf4a\") " pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:43.652583 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:43.652562    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:44.064717 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:44.064648    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d9llh"]
Oct 30 19:15:44.065976 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:44.065957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:44.066109 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:44.066085    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:44.066188 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:44.066156    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls podName:ffb9780a-2710-4742-b890-b7c0647a4086 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:45.06614226 +0000 UTC m=+32.606772961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls") pod "dns-default-2gdtt" (UID: "ffb9780a-2710-4742-b890-b7c0647a4086") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:44.067800 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:44.067780    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd392127_738d_43fd_8e0a_f6e207feaf4a.slice/crio-2105c8632598eaafad13b8a3a255da28408a79cba1c2ccfed85ed354966f33fb WatchSource:0}: Error finding container 2105c8632598eaafad13b8a3a255da28408a79cba1c2ccfed85ed354966f33fb: Status 404 returned error can't find the container with id 2105c8632598eaafad13b8a3a255da28408a79cba1c2ccfed85ed354966f33fb
Oct 30 19:15:44.218229 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:44.218199    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d9llh" event={"ID":"fd392127-738d-43fd-8e0a-f6e207feaf4a","Type":"ContainerStarted","Data":"2105c8632598eaafad13b8a3a255da28408a79cba1c2ccfed85ed354966f33fb"}
Oct 30 19:15:45.014640 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.014611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:45.014835 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.014646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:45.014835 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.014775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:45.067522 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.067495    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:15:45.068002 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.067525    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:15:45.072838 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.072792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:45.072935 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:45.072918    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:45.073002 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:45.072992    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls podName:ffb9780a-2710-4742-b890-b7c0647a4086 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:47.072973322 +0000 UTC m=+34.613604033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls") pod "dns-default-2gdtt" (UID: "ffb9780a-2710-4742-b890-b7c0647a4086") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:45.116995 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.116974    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:15:45.117949 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.117928    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gt8zl\""
Oct 30 19:15:45.118052 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.117951    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:15:45.118052 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.117932    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jlfgq\""
Oct 30 19:15:45.118052 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.117999    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-px7h6\""
Oct 30 19:15:45.878394 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.878359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:45.878567 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.878420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:45.878567 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:45.878537    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:15:45.878645 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:45.878619    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:16:17.878604135 +0000 UTC m=+65.419234849 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : secret "metrics-daemon-secret" not found
Oct 30 19:15:45.881083 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.881058    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:45.937047 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.937017    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:45.979074 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.979045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76vm\" (UniqueName: \"kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm\") pod \"network-check-target-qk5fg\" (UID: \"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519\") " pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:45.982438 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.982409    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n76vm\" (UniqueName: \"kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm\") pod \"network-check-target-qk5fg\" (UID: \"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519\") " pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:45.989189 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:45.989166    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:46.318651 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:46.318617    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n5gpj"]
Oct 30 19:15:46.363435 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:46.363408    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87fa4fa_f703_4fc3_ac9f_dc9a012bdf51.slice/crio-dd1174dc3c060015961d1072da6a486eb0b92d1586a7241421791c3284b80d38 WatchSource:0}: Error finding container dd1174dc3c060015961d1072da6a486eb0b92d1586a7241421791c3284b80d38: Status 404 returned error can't find the container with id dd1174dc3c060015961d1072da6a486eb0b92d1586a7241421791c3284b80d38
Oct 30 19:15:46.410948 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:46.410919    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qk5fg"]
Oct 30 19:15:46.414670 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:46.414646    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b32629_b4f3_4bb5_b8f2_ef5d0b634519.slice/crio-2b1619697a9486a8213b78fbb6ff9a9a0a473b2492a5f448b67b261761b71a82 WatchSource:0}: Error finding container 2b1619697a9486a8213b78fbb6ff9a9a0a473b2492a5f448b67b261761b71a82: Status 404 returned error can't find the container with id 2b1619697a9486a8213b78fbb6ff9a9a0a473b2492a5f448b67b261761b71a82
Oct 30 19:15:47.087335 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:47.087307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:47.087466 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:47.087428    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:47.087507 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:47.087477    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls podName:ffb9780a-2710-4742-b890-b7c0647a4086 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:51.087465539 +0000 UTC m=+38.628096251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls") pod "dns-default-2gdtt" (UID: "ffb9780a-2710-4742-b890-b7c0647a4086") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:47.226217 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:47.226183    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qk5fg" event={"ID":"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519","Type":"ContainerStarted","Data":"2b1619697a9486a8213b78fbb6ff9a9a0a473b2492a5f448b67b261761b71a82"}
Oct 30 19:15:47.227261 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:47.227237    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n5gpj" event={"ID":"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51","Type":"ContainerStarted","Data":"dd1174dc3c060015961d1072da6a486eb0b92d1586a7241421791c3284b80d38"}
Oct 30 19:15:48.231943 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:48.231902    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d9llh" event={"ID":"fd392127-738d-43fd-8e0a-f6e207feaf4a","Type":"ContainerStarted","Data":"57c345403bae115423c32aef5d1e24511d9a3fa4b39850ef351c10827363d246"}
Oct 30 19:15:49.511755 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.511693    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d9llh" podStartSLOduration=3.396224997 podStartE2EDuration="6.511675076s" totalImagesPullingTime="3.115450079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:48.396051439 +0000 UTC m=+35.936682161" watchObservedRunningTime="2026-10-30 19:15:49.511675076 +0000 UTC m=+37.052305798"
Oct 30 19:15:49.512258 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.511881    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d"]
Oct 30 19:15:49.539890 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.539540    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d"
Oct 30 19:15:49.574124 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.574087    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tqj9m"]
Oct 30 19:15:49.591617 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.591591    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d"]
Oct 30 19:15:49.591750 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.591694    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.609140 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.609110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fql4m\" (UniqueName: \"kubernetes.io/projected/d4008483-f3a6-4355-90df-db98415d8c2f-kube-api-access-fql4m\") pod \"migrator-66df5d496b-bv96d\" (UID: \"d4008483-f3a6-4355-90df-db98415d8c2f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d"
Oct 30 19:15:49.658848 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.658801    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tqj9m"]
Oct 30 19:15:49.706074 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.706049    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:15:49.706233 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.706137    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:15:49.706233 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.706180    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:15:49.706351 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.706314    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnhjf\""
Oct 30 19:15:49.706475 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.706457    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:49.706571 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.706516    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:15:49.709784 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.709763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fql4m\" (UniqueName: \"kubernetes.io/projected/d4008483-f3a6-4355-90df-db98415d8c2f-kube-api-access-fql4m\") pod \"migrator-66df5d496b-bv96d\" (UID: \"d4008483-f3a6-4355-90df-db98415d8c2f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d"
Oct 30 19:15:49.709897 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.709797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.709897 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.709869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/126d815b-b83a-4f35-a8ac-38f52d1c9a10-data-volume\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.710005 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.709905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/126d815b-b83a-4f35-a8ac-38f52d1c9a10-crio-socket\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.710005 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.709931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrzz\" (UniqueName: \"kubernetes.io/projected/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-api-access-ndrzz\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.710005 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.709977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.732082 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.732054    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4h2hd\""
Oct 30 19:15:49.732239 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.732214    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:15:49.767252 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.767192    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fql4m\" (UniqueName: \"kubernetes.io/projected/d4008483-f3a6-4355-90df-db98415d8c2f-kube-api-access-fql4m\") pod \"migrator-66df5d496b-bv96d\" (UID: \"d4008483-f3a6-4355-90df-db98415d8c2f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d"
Oct 30 19:15:49.811115 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.811049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/126d815b-b83a-4f35-a8ac-38f52d1c9a10-crio-socket\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.811115 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.811097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrzz\" (UniqueName: \"kubernetes.io/projected/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-api-access-ndrzz\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.811316 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.811151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.811316 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.811183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.811316 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.811229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/126d815b-b83a-4f35-a8ac-38f52d1c9a10-data-volume\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.811316 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.811271    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/126d815b-b83a-4f35-a8ac-38f52d1c9a10-crio-socket\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.811456 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:49.811361    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:49.811456 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:49.811407    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls podName:126d815b-b83a-4f35-a8ac-38f52d1c9a10 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:50.3113933 +0000 UTC m=+37.852024040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqj9m" (UID: "126d815b-b83a-4f35-a8ac-38f52d1c9a10") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:49.811561 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.811471    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/126d815b-b83a-4f35-a8ac-38f52d1c9a10-data-volume\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:49.811561 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:49.811524    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:15:49.811653 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:49.811578    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm podName:126d815b-b83a-4f35-a8ac-38f52d1c9a10 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:50.311566016 +0000 UTC m=+37.852196717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqj9m" (UID: "126d815b-b83a-4f35-a8ac-38f52d1c9a10") : configmap "kube-rbac-proxy" not found
Oct 30 19:15:49.867703 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.867645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d"
Oct 30 19:15:49.869672 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:49.869642    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrzz\" (UniqueName: \"kubernetes.io/projected/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-api-access-ndrzz\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:50.265383 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:50.265356    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d"]
Oct 30 19:15:50.270807 ip-10-0-133-7 kubenswrapper[2579]: W1030 19:15:50.270786    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4008483_f3a6_4355_90df_db98415d8c2f.slice/crio-588d1c8f004ab60a1410cac6d89ed7e52a904e28cd61c073828b2386429b6710 WatchSource:0}: Error finding container 588d1c8f004ab60a1410cac6d89ed7e52a904e28cd61c073828b2386429b6710: Status 404 returned error can't find the container with id 588d1c8f004ab60a1410cac6d89ed7e52a904e28cd61c073828b2386429b6710
Oct 30 19:15:50.313859 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:50.313829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:50.313980 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:50.313881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:50.313980 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:50.313970    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:15:50.314085 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:50.314005    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:50.314085 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:50.314030    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm podName:126d815b-b83a-4f35-a8ac-38f52d1c9a10 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:51.31401131 +0000 UTC m=+38.854642011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqj9m" (UID: "126d815b-b83a-4f35-a8ac-38f52d1c9a10") : configmap "kube-rbac-proxy" not found
Oct 30 19:15:50.314085 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:50.314068    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls podName:126d815b-b83a-4f35-a8ac-38f52d1c9a10 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:51.314049674 +0000 UTC m=+38.854680403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqj9m" (UID: "126d815b-b83a-4f35-a8ac-38f52d1c9a10") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:51.120449 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:51.120416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:51.120888 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:51.120541    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:51.120888 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:51.120605    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls podName:ffb9780a-2710-4742-b890-b7c0647a4086 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:59.120584684 +0000 UTC m=+46.661215401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls") pod "dns-default-2gdtt" (UID: "ffb9780a-2710-4742-b890-b7c0647a4086") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:51.242382 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:51.242343    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qk5fg" event={"ID":"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519","Type":"ContainerStarted","Data":"7f6a866a8c0f888f0b58007307f0dce67e385f748722567ea356950387a65ae6"}
Oct 30 19:15:51.243180 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:51.243114    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:51.244589 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:51.244562    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d" event={"ID":"d4008483-f3a6-4355-90df-db98415d8c2f","Type":"ContainerStarted","Data":"588d1c8f004ab60a1410cac6d89ed7e52a904e28cd61c073828b2386429b6710"}
Oct 30 19:15:51.321956 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:51.321927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:51.322150 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:51.322022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:51.322637 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:51.322594    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 19:15:51.322749 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:51.322674    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm podName:126d815b-b83a-4f35-a8ac-38f52d1c9a10 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:53.322653296 +0000 UTC m=+40.863284016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqj9m" (UID: "126d815b-b83a-4f35-a8ac-38f52d1c9a10") : configmap "kube-rbac-proxy" not found
Oct 30 19:15:51.322844 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:51.322751    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:51.322844 ip-10-0-133-7 kubenswrapper[2579]: E1030 19:15:51.322793    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls podName:126d815b-b83a-4f35-a8ac-38f52d1c9a10 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:53.322781849 +0000 UTC m=+40.863412550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqj9m" (UID: "126d815b-b83a-4f35-a8ac-38f52d1c9a10") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:51.358089 ip-10-0-133-7 kubenswrapper[2579]: I1030 19:15:51.358032    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qk5fg" podStartSLOduration=34.488640865 podStartE2EDuration="38.358013922s" totalImagesPullingTime="3.869373057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:51.355601595 +0000 UTC m=+38.896232318" watchObservedRunningTime="2026-10-30 19:15:51.358013922 +0000 UTC m=+38.898644644"
Oct 30 19:15:52.698790 ip-10-0-133-7 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 19:15:52.716153 ip-10-0-133-7 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 19:15:52.716315 ip-10-0-133-7 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 19:15:52.716360 ip-10-0-133-7 systemd[1]: kubelet.service: Consumed 2.147s CPU time.
Oct 30 19:15:52.726536 ip-10-0-133-7 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 19:15:52.816992 ip-10-0-133-7 kubenswrapper[4420]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:52.816992 ip-10-0-133-7 kubenswrapper[4420]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 19:15:52.816992 ip-10-0-133-7 kubenswrapper[4420]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:52.816992 ip-10-0-133-7 kubenswrapper[4420]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 19:15:52.818974 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.818906    4420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.818974 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.818975    4420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.818978    4420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.818981    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.818984    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.818986    4420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.818989    4420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.818991    4420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.818994    4420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.818996    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.818999    4420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819001    4420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819004    4420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819007    4420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819009    4420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819015    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819018    4420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819020    4420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819023    4420 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819025    4420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819027    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.819044 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819030    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819033    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819035    4420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819038    4420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819040    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819043    4420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819045    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819048    4420 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819050    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819053    4420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819056    4420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819059    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819061    4420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819064    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819067    4420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819069    4420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819072    4420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819074    4420 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819078    4420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.819522 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819082    4420 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819085    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819088    4420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819090    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819093    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819095    4420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819098    4420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819100    4420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819111    4420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819114    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819117    4420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819120    4420 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819122    4420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819125    4420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819127    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819130    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819132    4420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819135    4420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819137    4420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819139    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.819996 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819142    4420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819144    4420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819147    4420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819149    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819151    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819154    4420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819156    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819158    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819161    4420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819163    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819165    4420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819168    4420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819170    4420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819172    4420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819175    4420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819177    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819180    4420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819182    4420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819184    4420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.820472 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819187    4420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819190    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819200    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819203    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819205    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819208    4420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819210    4420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819213    4420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819215    4420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819217    4420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819220    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819222    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819224    4420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819227    4420 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819229    4420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819232    4420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819234    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819236    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819239    4420 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819243    4420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.820991 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819246    4420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819248    4420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819251    4420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819254    4420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819257    4420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819259    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.819262    4420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820000    4420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820007    4420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820009    4420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820012    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820014    4420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820017    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820020    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820024    4420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820027    4420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820030    4420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820032    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820035    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.821485 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820038    4420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820042    4420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820046    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820048    4420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820051    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820053    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820055    4420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820058    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820060    4420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820063    4420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820065    4420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820067    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820070    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820072    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820074    4420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820077    4420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820079    4420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820081    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820084    4420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.821965 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820086    4420 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820089    4420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820091    4420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820094    4420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820097    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820099    4420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820102    4420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820104    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820107    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820109    4420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820111    4420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820114    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820116    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820119    4420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820121    4420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820125    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820128    4420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820131    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820133    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.822423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820135    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820138    4420 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820140    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820142    4420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820145    4420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820147    4420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820149    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820151    4420 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820154    4420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820159    4420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820162    4420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820165    4420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820168    4420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820171    4420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820173    4420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820176    4420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820179    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820181    4420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820184    4420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820186    4420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.822894 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820189    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820191    4420 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820193    4420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820196    4420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820199    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820201    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820204    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820206    4420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820210    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820212    4420 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820215    4420 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820218    4420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820220    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820223    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820225    4420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820227    4420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820230    4420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820232    4420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820235    4420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820237    4420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.823357 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820240    4420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820242    4420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820244    4420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820247    4420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820249    4420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820253    4420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820256    4420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820258    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820260    4420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820263    4420 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820266    4420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820268    4420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820270    4420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820273    4420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820275    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820277    4420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820714    4420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820719    4420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820722    4420 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820724    4420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.823834 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820729    4420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820731    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820734    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820736    4420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820739    4420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820741    4420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820744    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820747    4420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820750    4420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820752    4420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820755    4420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820757    4420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820759    4420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820762    4420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820764    4420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820767    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820769    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820771    4420 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820774    4420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.824307 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820776    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820779    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820781    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820784    4420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820786    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820788    4420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820791    4420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820794    4420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820796    4420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820798    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820801    4420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820803    4420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820806    4420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820825    4420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820828    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820830    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820833    4420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820835    4420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820838    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820840    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.824789 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820844    4420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820846    4420 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820849    4420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820851    4420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820853    4420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820856    4420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820858    4420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820862    4420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820865    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820868    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820870    4420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820873    4420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820875    4420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820877    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820880    4420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820882    4420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820885    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820887    4420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820889    4420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.825289 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820892    4420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820894    4420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820896    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820899    4420 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820902    4420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820904    4420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820908    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820910    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820913    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820915    4420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820918    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820920    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820922    4420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820927    4420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820929    4420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820931    4420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820934    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820936    4420 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820940    4420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820942    4420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.825782 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820944    4420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820947    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820950    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820952    4420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820954    4420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820956    4420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820959    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820961    4420 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820964    4420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820966    4420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820969    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820973    4420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820977    4420 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820980    4420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820983    4420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820985    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820988    4420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820990    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820995    4420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.826275 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.820998    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821001    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821003    4420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821006    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821009    4420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821101    4420 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821113    4420 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821119    4420 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821123    4420 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821128    4420 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821132    4420 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821136    4420 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821139    4420 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821142    4420 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821145    4420 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821148    4420 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821151    4420 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821154    4420 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821157    4420 flags.go:64] FLAG: --cgroup-root=""
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821160    4420 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821163    4420 flags.go:64] FLAG: --client-ca-file=""
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821165    4420 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821168    4420 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 19:15:52.826767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821172    4420 flags.go:64] FLAG: --cluster-domain=""
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821175    4420 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821178    4420 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821180    4420 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821184    4420 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821187    4420 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821190    4420 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821193    4420 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821198    4420 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821201    4420 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821205    4420 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821207    4420 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821210    4420 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821213    4420 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821217    4420 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821221    4420 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821224    4420 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821226    4420 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821229    4420 flags.go:64] FLAG: --enable-server="true"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821232    4420 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821235    4420 flags.go:64] FLAG: --event-burst="100"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821238    4420 flags.go:64] FLAG: --event-qps="50"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821241    4420 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821244    4420 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 19:15:52.827644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821246    4420 flags.go:64] FLAG: --eviction-hard=""
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821250    4420 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821253    4420 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821256    4420 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821259    4420 flags.go:64] FLAG: --eviction-soft=""
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821262    4420 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821264    4420 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821267    4420 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821270    4420 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821272    4420 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821275    4420 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821278    4420 flags.go:64] FLAG: --feature-gates=""
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821282    4420 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821285    4420 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821287    4420 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821290    4420 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821294    4420 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821298    4420 flags.go:64] FLAG: --help="false"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821301    4420 flags.go:64] FLAG: --hostname-override="ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821304    4420 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821307    4420 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821309    4420 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821312    4420 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 19:15:52.828429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821320    4420 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821323    4420 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821325    4420 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821328    4420 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821330    4420 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821333    4420 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821336    4420 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821339    4420 flags.go:64] FLAG: --kube-reserved=""
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821342    4420 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821344    4420 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821347    4420 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821350    4420 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821352    4420 flags.go:64] FLAG: --lock-file=""
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821355    4420 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821358    4420 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821361    4420 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821366    4420 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821369    4420 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821371    4420 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821374    4420 flags.go:64] FLAG: --logging-format="text"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821377    4420 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821381    4420 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821384    4420 flags.go:64] FLAG: --manifest-url=""
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821386    4420 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821391    4420 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 19:15:52.828987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821395    4420 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821398    4420 flags.go:64] FLAG: --max-pods="110"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821402    4420 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821404    4420 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821407    4420 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821410    4420 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821412    4420 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821417    4420 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821420    4420 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821426    4420 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821429    4420 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821432    4420 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821435    4420 flags.go:64] FLAG: --pod-cidr=""
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821437    4420 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821440    4420 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821443    4420 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821445    4420 flags.go:64] FLAG: --port="10250"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821448    4420 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821451    4420 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-006473e1ec172c070"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821454    4420 flags.go:64] FLAG: --qos-reserved=""
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821457    4420 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821459    4420 flags.go:64] FLAG: --register-node="true"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821462    4420 flags.go:64] FLAG: --register-with-taints=""
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821466    4420 flags.go:64] FLAG: --registry-burst="10"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821469    4420 flags.go:64] FLAG: --registry-qps="5"
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821471    4420 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 19:15:52.829560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821474    4420 flags.go:64] FLAG: --reserved-memory=""
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821478    4420 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821481    4420 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821483    4420 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821486    4420 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821489    4420 flags.go:64] FLAG: --runonce="false"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821493    4420 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821496    4420 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821499    4420 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821502    4420 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821505    4420 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821508    4420 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821511    4420 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821515    4420 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821517    4420 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821520    4420 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821523    4420 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821526    4420 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821529    4420 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821531    4420 flags.go:64] FLAG: --system-cgroups=""
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821534    4420 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821539    4420 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821542    4420 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821545    4420 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821548    4420 flags.go:64] FLAG: --tls-min-version=""
Oct 30 19:15:52.830386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821550    4420 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821553    4420 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821556    4420 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821558    4420 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821561    4420 flags.go:64] FLAG: --v="2"
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821565    4420 flags.go:64] FLAG: --version="false"
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821568    4420 flags.go:64] FLAG: --vmodule=""
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821572    4420 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.821575    4420 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821705    4420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821709    4420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821712    4420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821714    4420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821719    4420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821722    4420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821724    4420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821726    4420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821729    4420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821732    4420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821734    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821738    4420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.831020 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821741    4420 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821743    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821746    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821748    4420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821750    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821753    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821755    4420 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821758    4420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821760    4420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821763    4420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821765    4420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821767    4420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821770    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821773    4420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821775    4420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821777    4420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821780    4420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821782    4420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821785    4420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821787    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.831521 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821789    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821792    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821794    4420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821797    4420 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821800    4420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821803    4420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821806    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821808    4420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821824    4420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821828    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821830    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821834    4420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821836    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821839    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821841    4420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821843    4420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821846    4420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821849    4420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821851    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821854    4420 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.832071 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821856    4420 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821858    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821861    4420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821863    4420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821865    4420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821868    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821870    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821873    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821876    4420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821879    4420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821882    4420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821885    4420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821887    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821890    4420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821893    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821895    4420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821899    4420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821901    4420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821904    4420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.832835 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821907    4420 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821909    4420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821912    4420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821915    4420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821918    4420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821921    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821923    4420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821926    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821928    4420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821930    4420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821933    4420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821936    4420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821939    4420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821942    4420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821945    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821947    4420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821950    4420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821952    4420 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821954    4420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821957    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.833406 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821959    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821962    4420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821964    4420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821966    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821969    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821971    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821974    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821976    4420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821979    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821982    4420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821984    4420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821986    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821989    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821991    4420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.821993    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.833973 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.822017    4420 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822049    4420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822053    4420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822055    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822058    4420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822060    4420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822063    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822065    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822067    4420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822070    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822072    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822074    4420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822076    4420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822079    4420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822081    4420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822084    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822086    4420 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822088    4420 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822091    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822093    4420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.834343 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822101    4420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822105    4420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822108    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822110    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822113    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822116    4420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822120    4420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822122    4420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822126    4420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822128    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822131    4420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822133    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822136    4420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822140    4420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822143    4420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822145    4420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822148    4420 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822150    4420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822153    4420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.834913 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822155    4420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822157    4420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822160    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822162    4420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822164    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822166    4420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822169    4420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822171    4420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822175    4420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822178    4420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822181    4420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822184    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822186    4420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822189    4420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822191    4420 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822193    4420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822196    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822198    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822200    4420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.835440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822203    4420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822205    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822208    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822210    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822213    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822215    4420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822217    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822221    4420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822223    4420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822226    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822228    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822231    4420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822233    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822235    4420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822237    4420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822240    4420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822242    4420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822244    4420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822247    4420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.835962 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822249    4420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822252    4420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822254    4420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822256    4420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822258    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822261    4420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822263    4420 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822265    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822268    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822270    4420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822272    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822275    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822277    4420 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822279    4420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822282    4420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822284    4420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822286    4420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822288    4420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822291    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822293    4420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.836467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822297    4420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822299    4420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822301    4420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822304    4420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822306    4420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822309    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822311    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822313    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822316    4420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.822318    4420 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.826555    4420 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.826572    4420 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.826591    4420 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.826596    4420 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827047    4420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827059    4420 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827064    4420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827075    4420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827080    4420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827085    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.836945 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827089    4420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827094    4420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827098    4420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827102    4420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827107    4420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827112    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827116    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827120    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827125    4420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827129    4420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827138    4420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827142    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827146    4420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827150    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827184    4420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827189    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827195    4420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827201    4420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827206    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.837440 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827210    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827214    4420 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827219    4420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827228    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827233    4420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827237    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827241    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827245    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827249    4420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827254    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827257    4420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827262    4420 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827266    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827270    4420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827274    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827278    4420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827287    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827291    4420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827294    4420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827298    4420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.837912 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827302    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827307    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827311    4420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827315    4420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827319    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827323    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827327    4420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827332    4420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827367    4420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827377    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827382    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827386    4420 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827392    4420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827398    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827432    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827437    4420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827441    4420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827445    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827449    4420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.838383 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827453    4420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827457    4420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827466    4420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827469    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827474    4420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827477    4420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827482    4420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827485    4420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827489    4420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827494    4420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827498    4420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827502    4420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827506    4420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827511    4420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827515    4420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827524    4420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827528    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827532    4420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827536    4420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.838969 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827539    4420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827543    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827547    4420 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827580    4420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827584    4420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827588    4420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827593    4420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827597    4420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827606    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827610    4420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827614    4420 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827618    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827622    4420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827627    4420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827631    4420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827644    4420 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827695    4420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827703    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827708    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.839467 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827713    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827719    4420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827724    4420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.827758    4420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.827957    4420 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828139    4420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828144    4420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828147    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828150    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828154    4420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828159    4420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828162    4420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828164    4420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828167    4420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828169    4420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.839926 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828172    4420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828175    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828177    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828180    4420 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828183    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828185    4420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828188    4420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828190    4420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828193    4420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828196    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828198    4420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828201    4420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828203    4420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828206    4420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828209    4420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828211    4420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828214    4420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828217    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828219    4420 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828221    4420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828224    4420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.840335 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828226    4420 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828228    4420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828231    4420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828234    4420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828236    4420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828238    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828241    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828243    4420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828246    4420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828248    4420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828252    4420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828255    4420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828258    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828261    4420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828263    4420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828266    4420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828269    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828271    4420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828274    4420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.840839 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828276    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828278    4420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828281    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828283    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828285    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828288    4420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828291    4420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828293    4420 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828296    4420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828298    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828301    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828303    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828305    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828308    4420 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828310    4420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828313    4420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828315    4420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828318    4420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828321    4420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.841342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828325    4420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828328    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828331    4420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828334    4420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828336    4420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828339    4420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828343    4420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828345    4420 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828348    4420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828351    4420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828354    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828356    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828359    4420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828361    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828364    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828366    4420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828369    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828371    4420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828373    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.841806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828376    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828379    4420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828381    4420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828384    4420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828386    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828388    4420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828391    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828393    4420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828396    4420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828398    4420 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828400    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828403    4420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828405    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828407    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828410    4420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828412    4420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828415    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.842272 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828418    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.828440    4420 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828470    4420 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828476    4420 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828479    4420 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828482    4420 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828485    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828487    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828490    4420 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828492    4420 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828495    4420 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828499    4420 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828502    4420 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828505    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828507    4420 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828510    4420 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 19:15:52.842685 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828512    4420 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828514    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828517    4420 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828519    4420 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828522    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828524    4420 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828526    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828529    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828531    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828533    4420 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828536    4420 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828538    4420 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828540    4420 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828543    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828545    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828548    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828550    4420 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828553    4420 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828555    4420 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828558    4420 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 19:15:52.843563 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828561    4420 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828563    4420 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828567    4420 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828571    4420 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828574    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828577    4420 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828579    4420 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828581    4420 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828584    4420 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828586    4420 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828589    4420 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828591    4420 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828593    4420 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828596    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828599    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828601    4420 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828604    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828606    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828609    4420 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 19:15:52.844201 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828611    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828613    4420 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828616    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828618    4420 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828620    4420 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828623    4420 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828625    4420 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828627    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828630    4420 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828632    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828635    4420 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828637    4420 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828639    4420 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828642    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828645    4420 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828647    4420 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828650    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828652    4420 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828654    4420 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 19:15:52.844658 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828657    4420 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828659    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828661    4420 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828664    4420 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828666    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828668    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828671    4420 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828673    4420 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828676    4420 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828678    4420 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828680    4420 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828683    4420 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828685    4420 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828688    4420 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828690    4420 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828692    4420 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828695    4420 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828697    4420 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828700    4420 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 19:15:52.845243 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828702    4420 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828704    4420 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828706    4420 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828709    4420 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828711    4420 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828714    4420 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828716    4420 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828719    4420 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828721    4420 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828724    4420 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828727    4420 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828729    4420 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828731    4420 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828733    4420 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:15:52.828736    4420 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.828789    4420 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.829697    4420 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.829750    4420 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.829964    4420 server.go:1014] "Starting client certificate rotation"
Oct 30 19:15:52.845717 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.830118    4420 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 19:15:52.846279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.830167    4420 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:10:12 +0000 UTC" deadline="2028-04-10 09:26:34.351374905 +0000 UTC"
Oct 30 19:15:52.846279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.830192    4420 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12662h10m41.521184271s"
Oct 30 19:15:52.846279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.830383    4420 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:15:52.846279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.831417    4420 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 19:15:52.846279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.833058    4420 log.go:25] "Validated CRI v1 runtime API"
Oct 30 19:15:52.846279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.834524    4420 log.go:25] "Validated CRI v1 image API"
Oct 30 19:15:52.846279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.835247    4420 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 19:15:52.846279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.836599    4420 fs.go:135] Filesystem UUIDs: map[197987f7-e5df-4815-b90a-6dbfaa81f174:/dev/nvme0n1p4 2c727091-369f-4aa9-b981-5a9d4a82d4c7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 30 19:15:52.846675 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.836617    4420 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2105c8632598eaafad13b8a3a255da28408a79cba1c2ccfed85ed354966f33fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2105c8632598eaafad13b8a3a255da28408a79cba1c2ccfed85ed354966f33fb/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2131acb0cb45e134376f22dc901d68cffaa7848b03f680182309b94cd65ddb74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2131acb0cb45e134376f22dc901d68cffaa7848b03f680182309b94cd65ddb74/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/22d296ffb312dfb38f65b32209805e059e6f47fcf36503b50fbfa6e6a809c8cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/22d296ffb312dfb38f65b32209805e059e6f47fcf36503b50fbfa6e6a809c8cb/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b1619697a9486a8213b78fbb6ff9a9a0a473b2492a5f448b67b261761b71a82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b1619697a9486a8213b78fbb6ff9a9a0a473b2492a5f448b67b261761b71a82/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/588d1c8f004ab60a1410cac6d89ed7e52a904e28cd61c073828b2386429b6710/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/588d1c8f004ab60a1410cac6d89ed7e52a904e28cd61c073828b2386429b6710/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a707cf7c9283c7172f415b55512868f664462a44150b95aaef1afdf6bbad56f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a707cf7c9283c7172f415b55512868f664462a44150b95aaef1afdf6bbad56f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8548a5f242ca23929b1fae7d7b633ab559c8d8332fc958cb61d548d44b322ba2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8548a5f242ca23929b1fae7d7b633ab559c8d8332fc958cb61d548d44b322ba2/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3a8254b28241ccbd5d240415df752f9cebe1cf720577e448c8d896820bf0ab5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3a8254b28241ccbd5d240415df752f9cebe1cf720577e448c8d896820bf0ab5/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a587ed1ea2bd536434d626e3121020adafb9ba0a664580fe92d66ae4d1d58fa6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a587ed1ea2bd536434d626e3121020adafb9ba0a664580fe92d66ae4d1d58fa6/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8040d9deb7b2993592593928dcb5f1a0ee9a9c499dfbcd7f198ab9689add99c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8040d9deb7b2993592593928dcb5f1a0ee9a9c499dfbcd7f198ab9689add99c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af3b653c88e7f39f1cc9e214dbeccfe70cef031b0bdf5d25678caa77179c4065/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af3b653c88e7f39f1cc9e214dbeccfe70cef031b0bdf5d25678caa77179c4065/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d128c72620004e303aa56724f53e24f277d58fecc5934d7d52538135626319de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d128c72620004e303aa56724f53e24f277d58fecc5934d7d52538135626319de/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd1174dc3c060015961d1072da6a486eb0b92d1586a7241421791c3284b80d38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd1174dc3c060015961d1072da6a486eb0b92d1586a7241421791c3284b80d38/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed2647f3f8def4c27969647f9054891824477ecaa703c6d0bae0cc9c7cd362b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed2647f3f8def4c27969647f9054891824477ecaa703c6d0bae0cc9c7cd362b3/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/093a7b2a-fb5b-4703-8073-98c60f1ef5ca/volumes/kubernetes.io~projected/kube-api-access-kms8h:{mountpoint:/var/lib/kubelet/pods/093a7b2a-fb5b-4703-8073-98c60f1ef5ca/volumes/kubernetes.io~projected/kube-api-access-kms8h major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/126d815b-b83a-4f35-a8ac-38f52d1c9a10/volumes/kubernetes.io~projected/kube-api-access-ndrzz:{mountpoint:/var/lib/kubelet/pods/126d815b-b83a-4f35-a8ac-38f52d1c9a10/volumes/kubernetes.io~projected/kube-api-access-ndrzz major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b725985-0e36-4850-b8ae-d761df575eb8/volumes/kubernetes.io~projected/kube-api-access-pv5l6:{mountpoint:/var/lib/kubelet/pods/2b725985-0e36-4850-b8ae-d761df575eb8/volumes/kubernetes.io~projected/kube-api-access-pv5l6 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30affd33-d7b4-4d40-a944-a2568dde24a0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/30affd33-d7b4-4d40-a944-a2568dde24a0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30affd33-d7b4-4d40-a944-a2568dde24a0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/30affd33-d7b4-4d40-a944-a2568dde24a0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30affd33-d7b4-4d40-a944-a2568dde24a0/volumes/kubernetes.io~projected/kube-api-access-vnkqf:{mountpoint:/var/lib/kubelet/pods/30affd33-d7b4-4d40-a944-a2568dde24a0/volumes/kubernetes.io~projected/kube-api-access-vnkqf major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/75c29e8f-e445-40b1-a2c0-1e8252b8f53f/volumes/kubernetes.io~projected/kube-api-access-lwkgq:{mountpoint:/var/lib/kubelet/pods/75c29e8f-e445-40b1-a2c0-1e8252b8f53f/volumes/kubernetes.io~projected/kube-api-access-lwkgq major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/866690aa-7dfb-4e2b-a29b-740a13dcc42e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/866690aa-7dfb-4e2b-a29b-740a13dcc42e/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98a8605a-fd55-4808-b74c-cd77b35fdbd3/volumes/kubernetes.io~projected/kube-api-access-jw57g:{mountpoint:/var/lib/kubelet/pods/98a8605a-fd55-4808-b74c-cd77b35fdbd3/volumes/kubernetes.io~projected/kube-api-access-jw57g major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d18034b-20ee-4e14-9ae9-7c680ff195a1/volumes/kubernetes.io~projected/kube-api-access-8lnrs:{mountpoint:/var/lib/kubelet/pods/9d18034b-20ee-4e14-9ae9-7c680ff195a1/volumes/kubernetes.io~projected/kube-api-access-8lnrs major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be/volumes/kubernetes.io~projected/kube-api-access-67pcp:{mountpoint:/var/lib/kubelet/pods/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be/volumes/kubernetes.io~projected/kube-api-access-67pcp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f/volumes/kubernetes.io~projected/kube-api-access-zgzpv:{mountpoint:/var/lib/kubelet/pods/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f/volumes/kubernetes.io~projected/kube-api-access-zgzpv major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519/volumes/kubernetes.io~projected/kube-api-access-n76vm:{mountpoint:/var/lib/kubelet/pods/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519/volumes/kubernetes.io~projected/kube-api-access-n76vm major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4008483-f3a6-4355-90df-db98415d8c2f/volumes/kubernetes.io~projected/kube-api-access-fql4m:{mountpoint:/var/lib/kubelet/pods/d4008483-f3a6-4355-90df-db98415d8c2f/volumes/kubernetes.io~projected/kube-api-access-fql4m major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac/volumes/kubernetes.io~projected/kube-api-access-j7pq5:{mountpoint:/var/lib/kubelet/pods/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac/volumes/kubernetes.io~projected/kube-api-access-j7pq5 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd392127-738d-43fd-8e0a-f6e207feaf4a/volumes/kubernetes.io~projected/kube-api-access-wgz84:{mountpoint:/var/lib/kubelet/pods/fd392127-738d-43fd-8e0a-f6e207feaf4a/volumes/kubernetes.io~projected/kube-api-access-wgz84 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd392127-738d-43fd-8e0a-f6e207feaf4a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/fd392127-738d-43fd-8e0a-f6e207feaf4a/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb9780a-2710-4742-b890-b7c0647a4086/volumes/kubernetes.io~projected/kube-api-access-vn4zj:{mountpoint:/var/lib/kubelet/pods/ffb9780a-2710-4742-b890-b7c0647a4086/volumes/kubernetes.io~projected/kube-api-access-vn4zj major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/7edee5d8ba39bfb0f199958dc7eb990fde493a1e4f9f73b9bbdec18d26f20009/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/6f790e675b4abaa4233aa0ce96682e30cd2277b8564acc11fc5e0f9b41da7f51/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/eab239864306e926b892e75a96d6e0c747d0c79b1daf3078e983d08a6e4731ca/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ffea15089d49991fccdb037e88967edd32abcf7bd2b61b6930ee85f909763a9f/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/bdbb020b5de2ef27adafe5707fc458396d927a88b458ec38f6798d6130d4eda5/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f6c294a3c44c8f7ea99c9eb10f811f975603264b13a308f290d01f6410527329/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/9a0cebd3a7f9c9ae48dc6e09509658eb8dfb7fd2bdb1fa4a8c0895fdab07045d/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/76db9ff5d916d1b52e23c90c391270f92741220f48af34c62baa56b10c70a0d7/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d1ffdf9d7e7a1a0557371c4afc3034f7a99722d911dc02aafd68f6928fbdb973/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/ff30fab1da918956d9abda67302abbdd87781bb2fad72e6e6979722e1f708c49/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/981acd4f40c044c781530bcca76a50fa50aed56a208d8f2482263eeeb2f2fac1/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/8e907fec093eb73bcbb33f9c3d662dc4e86b21f824ccb486bd53ee8de4660201/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1df5dbe9b7a4b06ae227b1ae976de4225124aa9a9c3c4f16b99d8e5475f81619/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/d51dfddae0b8b5e6ccb837de41aa6dc4af5cd34d34b9631a04d4327ab7d959b9/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/6818c56f973b14d672d45d9d284116c5d614ac1c105a9fd4a1a2d1c8beb9f045/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/bacd5d3b91245932e1c1a3da9ad79117c0ddc0a9661b52b3716f96aa77fcaac0/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6aadf34a35207ccbeb4f79bd09b20d56cfa2cd22845f79e7e84bfa5c9ea2dba0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/aa0424c60f4ce33b294b67aa4aa947488460d61c4da562f22f6c0a4b3fe1a3e2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2c1babd3cdfdd0cb16f674ca9d03356736ee62e02590295f2210d798b0cb244d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/3b9a486fc39f2a404df75f3321e947e7a67872d446b704af35a0a96720debbcd/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a2bbb99a574a07e5de4d6ef087613bad158dbf5d8913a51133857955ef98202a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/b0bbeb0ffb63dfa948ac2eb0d8fadab68eeb157e9dbdf28080588039c530d462/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/4b7c7a02dfa426b4a3a3e636a5fa3cfcc3949e9f7a36780256da5873ea87f309/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/c68ae06e1a5bb31740e1c4434737f6eb942c2cf9a1524da89301d9cabb2df1b0/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/9a634a0443375b053457f30f408149690fc5bbada74be6bf54ec49f9eb3ccb3f/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/0de367881e2a3ce5333020607d468954534295fe9e054a4d8d692d131c42ad8f/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/24f0494a3e588c3ed9a51ee2ccbac64429210b40de21b7b9597c6226c602d1b4/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/331a228edf9dd1c086198971db775d8d9ca0eeea6f56c06f1ed3a0b13f6a7e23/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/193792dbcd6b43d656ea650cec9a9cf7305a5f7eb25694ecabab2618d4f878d3/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/9a6e7342fe3014c0c9f4fab92e226f92d575595360cb01b45d14f3062629e924/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/369a54b755dc6f4fa379df9fbb5707f873b8b409f7b11d9fe9581ed2dcffd144/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0af05d86774bdfa0fa16ac6b855c39e9aac684b600550c73c91a3b4f3b99b6c2/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/6133e12bcbe3f250e9139ec2f6b6cac192d241f549ea639eef0b17ab66fa0538/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/ce18189d727e0f71d5231c5edce476e27461a975f1679d52016113693251531d/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/99e42498211d81a4a18dff4680a9cafe0b982285aaad770f9d0421f46851f855/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/bfac12ac3889bc765dbc0d7d1fd112456f71c5178dbba8b370f61532a2514a6c/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/1876c39e9e579b4d5cb95a04af6f8993757968d17e6019f725c2dfae3ee0a184/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/9aaafc41fc920eda6545f662ab502806e02c89a86f6971a86418f838b71193e2/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b1979a32a75dece80d5c68821e353c347b62c8a6dee48b5da63e56e783d0505b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.842956    4420 manager.go:217] Machine: {Timestamp:2026-10-30 19:15:52.842081783 +0000 UTC m=+0.053298937 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3203138 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24bb9cf4f80b2cda16eff606ccc982 SystemUUID:ec24bb9c-f4f8-0b2c-da16-eff606ccc982 BootID:84af1f8f-1a5b-44e3-bb18-aa04c43351cd Filesystems:[{Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/30affd33-d7b4-4d40-a944-a2568dde24a0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:48 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3a8254b28241ccbd5d240415df752f9cebe1cf720577e448c8d896820bf0ab5/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a707cf7c9283c7172f415b55512868f664462a44150b95aaef1afdf6bbad56f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be/volumes/kubernetes.io~projected/kube-api-access-67pcp DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/093a7b2a-fb5b-4703-8073-98c60f1ef5ca/volumes/kubernetes.io~projected/kube-api-access-kms8h DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2131acb0cb45e134376f22dc901d68cffaa7848b03f680182309b94cd65ddb74/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed2647f3f8def4c27969647f9054891824477ecaa703c6d0bae0cc9c7cd362b3/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb9780a-2710-4742-b890-b7c0647a4086/volumes/kubernetes.io~projected/kube-api-access-vn4zj DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8040d9deb7b2993592593928dcb5f1a0ee9a9c499dfbcd7f198ab9689add99c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8548a5f242ca23929b1fae7d7b633ab559c8d8332fc958cb61d548d44b322ba2/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/fd392127-738d-43fd-8e0a-f6e207feaf4a/volumes/kubernetes.io~projected/kube-api-access-wgz84 DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d128c72620004e303aa56724f53e24f277d58fecc5934d7d52538135626319de/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/126d815b-b83a-4f35-a8ac-38f52d1c9a10/volumes/kubernetes.io~projected/kube-api-access-ndrzz DeviceMajor:0 DeviceMinor:223 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/30affd33-d7b4-4d40-a944-a2568dde24a0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/98a8605a-fd55-4808-b74c-cd77b35fdbd3/volumes/kubernetes.io~projected/kube-api-access-jw57g DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519/volumes/kubernetes.io~projected/kube-api-access-n76vm DeviceMajor:0 DeviceMinor:209 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d4008483-f3a6-4355-90df-db98415d8c2f/volumes/kubernetes.io~projected/kube-api-access-fql4m DeviceMajor:0 DeviceMinor:218 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd392127-738d-43fd-8e0a-f6e207feaf4a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd1174dc3c060015961d1072da6a486eb0b92d1586a7241421791c3284b80d38/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac/volumes/kubernetes.io~projected/kube-api-access-j7pq5 DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2105c8632598eaafad13b8a3a255da28408a79cba1c2ccfed85ed354966f33fb/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b1619697a9486a8213b78fbb6ff9a9a0a473b2492a5f448b67b261761b71a82/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/75c29e8f-e445-40b1-a2c0-1e8252b8f53f/volumes/kubernetes.io~projected/kube-api-access-lwkgq DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af3b653c88e7f39f1cc9e214dbeccfe70cef031b0bdf5d25678caa77179c4065/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/22d296ffb312dfb38f65b32209805e059e6f47fcf36503b50fbfa6e6a809c8cb/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/588d1c8f004ab60a1410cac6d89ed7e52a904e28cd61c073828b2386429b6710/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/30affd33-d7b4-4d40-a944-a2568dde24a0/volumes/kubernetes.io~projected/kube-api-access-vnkqf DeviceMajor:0 DeviceMinor:52 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f/volumes/kubernetes.io~projected/kube-api-access-zgzpv DeviceMajor:0 DeviceMinor:53 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b725985-0e36-4850-b8ae-d761df575eb8/volumes/kubernetes.io~projected/kube-api-access-pv5l6 DeviceMajor:0 DeviceMinor:54 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/866690aa-7dfb-4e2b-a29b-740a13dcc42e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9d18034b-20ee-4e14-9ae9-7c680ff195a1/volumes/kubernetes.io~projected/kube-api-access-8lnrs DeviceMajor:0 DeviceMinor:51 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a587ed1ea2bd536434d626e3121020adafb9ba0a664580fe92d66ae4d1d58fa6/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2105c8632598eaa MacAddress:3a:5b:85:84:ca:d3 Speed:10000 Mtu:8901} {Name:2b1619697a9486a MacAddress:ea:4f:84:51:95:85 Speed:10000 Mtu:8901} {Name:588d1c8f004ab60 MacAddress:ce:4c:df:be:72:e5 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:58:ce:ce:97:81 Speed:0 Mtu:9001} {Name:br-int MacAddress:fa:fc:94:ad:3e:3a Speed:0 Mtu:8901} {Name:dd1174dc3c06001 MacAddress:9e:56:b0:73:3f:13 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:58:ce:ce:97:81 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e6:00:af:ea:8d:d1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:82:ef:52:5a:b7:0d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.843646    4420 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.843730    4420 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.844502    4420 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.844524    4420 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-7.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.844636    4420 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.844643    4420 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.844657    4420 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.844676    4420 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.844752    4420 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.844956    4420 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.845066    4420 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.845102    4420 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.845117    4420 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.845127    4420 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.845135    4420 kubelet.go:405] "Adding apiserver pod source"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.845142    4420 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.845784    4420 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.845793    4420 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:15:52.847065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.846298    4420 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 19:15:52.848689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.848670    4420 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 19:15:52.848689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.848688    4420 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 19:15:52.848803 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.848792    4420 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 19:15:52.849037 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849021    4420 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 19:15:52.849098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849060    4420 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 19:15:52.849215 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849203    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 19:15:52.849268 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849225    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 19:15:52.849268 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849235    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 19:15:52.849268 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849244    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 19:15:52.849268 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849253    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 19:15:52.849268 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849263    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 19:15:52.849498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849271    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 19:15:52.849498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849282    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 19:15:52.849498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849291    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 19:15:52.849498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849300    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 19:15:52.849498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849312    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 19:15:52.849498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849325    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 19:15:52.849498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849359    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 19:15:52.849498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.849369    4420 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 19:15:52.852615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.852599    4420 server.go:1263] "Started kubelet"
Oct 30 19:15:52.852753 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.852682    4420 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 19:15:52.852906 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.852831    4420 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 19:15:52.852906 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.852901    4420 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 19:15:52.853144 ip-10-0-133-7 systemd[1]: Started Kubernetes Kubelet.
Oct 30 19:15:52.853928 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.853867    4420 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 19:15:52.854659 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.854645    4420 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.856461    4420 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.856547    4420 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.856606    4420 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.856618    4420 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.856700    4420 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.857350    4420 factory.go:55] Registering systemd factory
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.857370    4420 factory.go:223] Registration of the systemd container factory successfully
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.857885    4420 factory.go:153] Registering CRI-O factory
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.857899    4420 factory.go:223] Registration of the crio container factory successfully
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.857962    4420 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.857980    4420 factory.go:103] Registering Raw factory
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.857994    4420 manager.go:1199] Started watching for new ooms in manager
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858321    4420 manager.go:319] Starting recovery of all containers
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858756    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="093a7b2a-fb5b-4703-8073-98c60f1ef5ca" volumeName="kubernetes.io/projected/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-kube-api-access-kms8h" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858800    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be" volumeName="kubernetes.io/secret/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858835    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb9780a-2710-4742-b890-b7c0647a4086" volumeName="kubernetes.io/empty-dir/ffb9780a-2710-4742-b890-b7c0647a4086-tmp-dir" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858847    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51" volumeName="kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858857    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd392127-738d-43fd-8e0a-f6e207feaf4a" volumeName="kubernetes.io/secret/fd392127-738d-43fd-8e0a-f6e207feaf4a-cert" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858866    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4b32629-b4f3-4bb5-b8f2-ef5d0b634519" volumeName="kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858876    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75c29e8f-e445-40b1-a2c0-1e8252b8f53f" volumeName="kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-daemon-config" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858885    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75c29e8f-e445-40b1-a2c0-1e8252b8f53f" volumeName="kubernetes.io/projected/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-kube-api-access-lwkgq" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858896    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="093a7b2a-fb5b-4703-8073-98c60f1ef5ca" volumeName="kubernetes.io/configmap/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-serviceca" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858907    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be" volumeName="kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858916    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2d3697f-f464-4a7d-af17-4ea6d73ae5ac" volumeName="kubernetes.io/empty-dir/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-tmp-dir" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858926    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d18034b-20ee-4e14-9ae9-7c680ff195a1" volumeName="kubernetes.io/configmap/9d18034b-20ee-4e14-9ae9-7c680ff195a1-iptables-alerter-script" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858937    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be" volumeName="kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-config" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858963    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2d3697f-f464-4a7d-af17-4ea6d73ae5ac" volumeName="kubernetes.io/projected/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-kube-api-access-j7pq5" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858976    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb9780a-2710-4742-b890-b7c0647a4086" volumeName="kubernetes.io/projected/ffb9780a-2710-4742-b890-b7c0647a4086-kube-api-access-vn4zj" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.858991    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="126d815b-b83a-4f35-a8ac-38f52d1c9a10" volumeName="kubernetes.io/empty-dir/126d815b-b83a-4f35-a8ac-38f52d1c9a10-data-volume" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859003    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="126d815b-b83a-4f35-a8ac-38f52d1c9a10" volumeName="kubernetes.io/projected/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-api-access-ndrzz" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859014    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b725985-0e36-4850-b8ae-d761df575eb8" volumeName="kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859025    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30affd33-d7b4-4d40-a944-a2568dde24a0" volumeName="kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-tuned" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859036    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="866690aa-7dfb-4e2b-a29b-740a13dcc42e" volumeName="kubernetes.io/secret/866690aa-7dfb-4e2b-a29b-740a13dcc42e-agent-certs" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859049    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a8605a-fd55-4808-b74c-cd77b35fdbd3" volumeName="kubernetes.io/projected/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kube-api-access-jw57g" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859060    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4008483-f3a6-4355-90df-db98415d8c2f" volumeName="kubernetes.io/projected/d4008483-f3a6-4355-90df-db98415d8c2f-kube-api-access-fql4m" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859072    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb9780a-2710-4742-b890-b7c0647a4086" volumeName="kubernetes.io/configmap/ffb9780a-2710-4742-b890-b7c0647a4086-config-volume" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859084    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30affd33-d7b4-4d40-a944-a2568dde24a0" volumeName="kubernetes.io/projected/30affd33-d7b4-4d40-a944-a2568dde24a0-kube-api-access-vnkqf" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859095    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="75c29e8f-e445-40b1-a2c0-1e8252b8f53f" volumeName="kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859105    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="866690aa-7dfb-4e2b-a29b-740a13dcc42e" volumeName="kubernetes.io/configmap/866690aa-7dfb-4e2b-a29b-740a13dcc42e-konnectivity-ca" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859115    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd392127-738d-43fd-8e0a-f6e207feaf4a" volumeName="kubernetes.io/projected/fd392127-738d-43fd-8e0a-f6e207feaf4a-kube-api-access-wgz84" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859134    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b725985-0e36-4850-b8ae-d761df575eb8" volumeName="kubernetes.io/projected/2b725985-0e36-4850-b8ae-d761df575eb8-kube-api-access-pv5l6" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859149    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30affd33-d7b4-4d40-a944-a2568dde24a0" volumeName="kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-tmp" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859161    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be" volumeName="kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-env-overrides" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859172    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be" volumeName="kubernetes.io/projected/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-kube-api-access-67pcp" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859182    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b725985-0e36-4850-b8ae-d761df575eb8" volumeName="kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859200    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b725985-0e36-4850-b8ae-d761df575eb8" volumeName="kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-binary-copy" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859211    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d18034b-20ee-4e14-9ae9-7c680ff195a1" volumeName="kubernetes.io/projected/9d18034b-20ee-4e14-9ae9-7c680ff195a1-kube-api-access-8lnrs" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859221    4420 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c36c31f3-8f2c-4201-b342-9d5c5bab7b6f" volumeName="kubernetes.io/projected/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-kube-api-access-zgzpv" seLinuxMountContext=""
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859233    4420 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.859239    4420 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 19:15:52.858357 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:52.859587    4420 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-7.us-west-1.compute.internal\" not found"
Oct 30 19:15:52.862971 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.861584    4420 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:10:13 +0000 UTC" deadline="2028-05-13 20:17:29.383204142 +0000 UTC"
Oct 30 19:15:52.862971 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.861618    4420 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13465h1m36.521588411s"
Oct 30 19:15:52.886264 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.886229    4420 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 19:15:52.888012 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.887990    4420 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 19:15:52.888012 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.888014    4420 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 19:15:52.888199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.888033    4420 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 19:15:52.888199 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:52.888070    4420 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 19:15:52.900330 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.900304    4420 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="a8c4ce81a2d4d47e9100a03473e7258ec653aa6173bb4b9203c22197a735f8b4" exitCode=0
Oct 30 19:15:52.900330 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.900324    4420 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="1fa259238fc7bfed4e5b04995e44bd17f2f3ed47e901fcaf7349e099409901c5" exitCode=0
Oct 30 19:15:52.900509 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.900333    4420 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="04799b3026c89376f49d3236c697d47ca00ae4783e87e00fe1b3f3e31afb29d0" exitCode=0
Oct 30 19:15:52.900509 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.900343    4420 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="5f8e9bafd05ee42b7169a28a96a8f9be885c79ceb673f1a913aba2bf8bfd9b60" exitCode=0
Oct 30 19:15:52.900509 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.900364    4420 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="04d64e64975c16ea629d1f9847ed4cf1768f60e9fe788643ab101e9370f35815" exitCode=0
Oct 30 19:15:52.900509 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.900371    4420 generic.go:356] "Generic (PLEG): container finished" podID="2b725985-0e36-4850-b8ae-d761df575eb8" containerID="7083dd6cc684392f5c4d01ec21a92eec95baaa44b62b0d8efb416bb8dbcdea9f" exitCode=0
Oct 30 19:15:52.913034 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.913009    4420 generic.go:356] "Generic (PLEG): container finished" podID="95d302d841e966ec6a147c94830317a7" containerID="3f0830ec228c8c1422f267d8e79c4542e529fbcfe83318f742ccce8169b89c08" exitCode=0
Oct 30 19:15:52.922159 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.922144    4420 manager.go:324] Recovery completed
Oct 30 19:15:52.934122 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.934092    4420 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.935994 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.935976    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.936073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.936005    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.936073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.936015    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.937218 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937204    4420 cpu_manager.go:226] "Starting" policy="none"
Oct 30 19:15:52.937218 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937217    4420 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 19:15:52.937295 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937224    4420 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.937295 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937232    4420 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 19:15:52.937360 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937334    4420 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 19:15:52.937360 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937344    4420 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 19:15:52.937360 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937359    4420 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 19:15:52.937498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937364    4420 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 19:15:52.937498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937370    4420 policy_none.go:50] "Start"
Oct 30 19:15:52.937498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937380    4420 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 19:15:52.937498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937389    4420 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 19:15:52.937619 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937503    4420 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 19:15:52.937619 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937509    4420 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 19:15:52.937619 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937515    4420 policy_none.go:44] "Start"
Oct 30 19:15:52.937934 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937919    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.938009 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937951    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.938009 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.937968    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.941525 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.941502    4420 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 19:15:52.944121 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.944109    4420 manager.go:341] "Starting Device Plugin manager"
Oct 30 19:15:52.944191 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:52.944132    4420 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 19:15:52.944191 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.944148    4420 server.go:87] "Starting device plugin registration server"
Oct 30 19:15:52.944380 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.944368    4420 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 19:15:52.944485 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.944380    4420 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 19:15:52.944527 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.944486    4420 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 19:15:52.944596 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.944569    4420 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 19:15:52.944643 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.944611    4420 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 19:15:52.944643 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.944618    4420 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 19:15:52.944744 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.944658    4420 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:15:52.944595204Z","UUID":"490f22dd-f0d4-4ffe-a90a-1552e7fc4fe5","Handler":null,"Name":"","Endpoint":""}
Oct 30 19:15:52.945983 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.945970    4420 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 19:15:52.945983 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.945991    4420 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 19:15:52.955276 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:52.955220    4420 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 19:15:52.955276 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:52.955262    4420 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-7.us-west-1.compute.internal\" not found"
Oct 30 19:15:52.989020 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.988588    4420 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"]
Oct 30 19:15:52.989110 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.989100    4420 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.992786 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.992770    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.992877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.992803    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.992877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.992845    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.993084 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.993073    4420 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.993221 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.993209    4420 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.993839 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.993801    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.993923 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.993843    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.993923 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.993855    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.993923 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.993869    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.993923 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.993880    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.993923 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.993870    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.994071 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:52.994056    4420 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-7.us-west-1.compute.internal\" not found" node="ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:52.994121 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.994089    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal" event={"ID":"67762efed53c0ad4c56b95d172cd37b4","Type":"ContainerStarted","Data":"e4ffb23a4a050f7a6a61b2c16f7769c73038394c5bb0da3abb9367a6d263f734"}
Oct 30 19:15:52.994162 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.994129    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal" event={"ID":"67762efed53c0ad4c56b95d172cd37b4","Type":"ContainerStarted","Data":"d128c72620004e303aa56724f53e24f277d58fecc5934d7d52538135626319de"}
Oct 30 19:15:52.994192 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.994178    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal" event={"ID":"95d302d841e966ec6a147c94830317a7","Type":"ContainerStarted","Data":"e88e54d6b3d66d91754a3d9294bc73a3029b25ae773520efbb2af234c90c52e7"}
Oct 30 19:15:52.994192 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.994189    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal" event={"ID":"95d302d841e966ec6a147c94830317a7","Type":"ContainerDied","Data":"3f0830ec228c8c1422f267d8e79c4542e529fbcfe83318f742ccce8169b89c08"}
Oct 30 19:15:52.994251 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.994195    4420 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:52.994280 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.994199    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal" event={"ID":"95d302d841e966ec6a147c94830317a7","Type":"ContainerStarted","Data":"af3b653c88e7f39f1cc9e214dbeccfe70cef031b0bdf5d25678caa77179c4065"}
Oct 30 19:15:52.994931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.994918    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:52.994994 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.994939    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:52.994994 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:52.994953    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:52.995076 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:52.995066    4420 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-7.us-west-1.compute.internal\" not found" node="ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.044456 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.044439    4420 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 19:15:53.045163 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.045147    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 19:15:53.045254 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.045171    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 19:15:53.045254 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.045183    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 19:15:53.045254 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.045199    4420 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.048747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.048734    4420 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 19:15:53.050622 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.050607    4420 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:53.050688 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.050676    4420 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:53.063208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.063195    4420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.100275 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.100253    4420 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:53.100566 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.100547    4420 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 19:15:53.147899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.147884    4420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:53.147963 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:53.147913    4420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.147963 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.147922    4420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.149000 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.148975    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.149072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.149009    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.149072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.149032    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67762efed53c0ad4c56b95d172cd37b4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"67762efed53c0ad4c56b95d172cd37b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.156619 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.156598    4420 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.156689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.156655    4420 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.157049 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.157036    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 19:15:53.157097 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.157061    4420 setters.go:546] "Node became not ready" node="ip-10-0-133-7.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T19:15:53Z","lastTransitionTime":"2026-10-30T19:15:53Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 19:15:53.206018 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.205974    4420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:53.206018 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:53.206009    4420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.214065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.214049    4420 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-7.us-west-1.compute.internal" is owned by different node (oldNodeID="d08f7344-1a7d-4c77-82b6-36ad27b3d7c7", newNodeID=""), cleaning up...
Oct 30 19:15:53.249563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.249541    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.249653 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.249568    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.249653 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.249586    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67762efed53c0ad4c56b95d172cd37b4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"67762efed53c0ad4c56b95d172cd37b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.249653 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.249619    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/67762efed53c0ad4c56b95d172cd37b4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"67762efed53c0ad4c56b95d172cd37b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.249653 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.249646    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.249790 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.249666    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95d302d841e966ec6a147c94830317a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" (UID: \"95d302d841e966ec6a147c94830317a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.845946 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.845922    4420 apiserver.go:52] "Watching apiserver"
Oct 30 19:15:53.923590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.923567    4420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.923737 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.923721    4420 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.948800 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.948775    4420 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 19:15:53.949222 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.949202    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-n5gpj","openshift-dns/dns-default-2gdtt","kube-system/konnectivity-agent-sdtv7","openshift-multus/multus-hr8dq","openshift-dns/node-resolver-x2pz6","openshift-multus/multus-additional-cni-plugins-475ns","openshift-cluster-node-tuning-operator/tuned-n7llw","openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d","openshift-network-operator/iptables-alerter-dx9vb","openshift-ovn-kubernetes/ovnkube-node-vspvz","openshift-insights/insights-runtime-extractor-tqj9m","openshift-ingress-canary/ingress-canary-d9llh","openshift-image-registry/node-ca-hlnt2","openshift-multus/network-metrics-daemon-gwzbc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8","openshift-network-diagnostics/network-check-target-qk5fg"]
Oct 30 19:15:53.949906 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.949886    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:53.953186 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.953156    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:53.955421 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.955397    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:53.958328 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.958311    4420 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 19:15:53.984335 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.984318    4420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:53.984397 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:53.984355    4420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:53.984871 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:53.984860    4420 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 19:15:53.984906 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:53.984886    4420 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-7.us-west-1.compute.internal"
Oct 30 19:15:54.002734 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.002716    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 19:15:54.002845 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.002733    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5vtr8\""
Oct 30 19:15:54.002845 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.002750    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 19:15:54.002981 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.002966    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 19:15:54.003041 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.002992    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 19:15:54.003127 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.003113    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:54.003179 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.003111    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 19:15:54.003179 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.003118    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 19:15:54.003179 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.003120    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:54.003179 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.003175    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 19:15:54.052924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.052901    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovn-node-metrics-cert\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.053003 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.052935    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.053003 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.052958    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-tmp-dir\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:54.053003 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.052982    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.053120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053032    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.053120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053055    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-ovn\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.053120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053070    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-log-socket\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.053120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053094    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-tmp-dir\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:54.053120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053110    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-systemd\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.053290 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053138    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.053290 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053171    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.053290 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053191    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-sys-fs\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.053290 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053221    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd392127-738d-43fd-8e0a-f6e207feaf4a-cert\") pod \"ingress-canary-d9llh\" (UID: \"fd392127-738d-43fd-8e0a-f6e207feaf4a\") " pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:54.053400 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053321    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5l6\" (UniqueName: \"kubernetes.io/projected/2b725985-0e36-4850-b8ae-d761df575eb8-kube-api-access-pv5l6\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.053400 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053344    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-script-lib\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.053400 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053390    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-sys\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.053500 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053481    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-binary-copy\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.053537 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053498    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-tuned\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.053537 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053517    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76vm\" (UniqueName: \"kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm\") pod \"network-check-target-qk5fg\" (UID: \"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519\") " pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:54.053589 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053534    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-registration-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.053589 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053557    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d18034b-20ee-4e14-9ae9-7c680ff195a1-host-slash\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:54.053589 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053557    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-tuned\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.053589 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053572    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fql4m\" (UniqueName: \"kubernetes.io/projected/d4008483-f3a6-4355-90df-db98415d8c2f-kube-api-access-fql4m\") pod \"migrator-66df5d496b-bv96d\" (UID: \"d4008483-f3a6-4355-90df-db98415d8c2f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d"
Oct 30 19:15:54.053701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053604    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-bin\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.053701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053628    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-env-overrides\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.053701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053652    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgzpv\" (UniqueName: \"kubernetes.io/projected/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-kube-api-access-zgzpv\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:54.053790 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053707    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cni-binary-copy\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.053790 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053733    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.053790 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053759    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-netns\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.053897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053789    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-systemd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.053897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053854    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.053897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053885    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysconfig\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.053976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053906    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-kubelet\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.053976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053922    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-etc-kubernetes\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.053976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053956    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9d18034b-20ee-4e14-9ae9-7c680ff195a1-iptables-alerter-script\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:54.054058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053986    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-conf\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.054058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.053989    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cni-binary-copy\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.054058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054051    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-system-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.054154 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054070    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-conf-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.054154 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054083    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-dbus\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:54.054154 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054119    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:54.054242 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054154    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb9780a-2710-4742-b890-b7c0647a4086-tmp-dir\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:54.054242 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054171    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/126d815b-b83a-4f35-a8ac-38f52d1c9a10-crio-socket\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.054242 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054171    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9d18034b-20ee-4e14-9ae9-7c680ff195a1-iptables-alerter-script\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:54.054242 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054187    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-k8s-cni-cncf-io\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.054350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054271    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-multus\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.054350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054272    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffb9780a-2710-4742-b890-b7c0647a4086-tmp-dir\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:54.054350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054300    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb9780a-2710-4742-b890-b7c0647a4086-config-volume\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:54.054350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054338    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-netd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.054458 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054371    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.054458 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054395    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/866690aa-7dfb-4e2b-a29b-740a13dcc42e-konnectivity-ca\") pod \"konnectivity-agent-sdtv7\" (UID: \"866690aa-7dfb-4e2b-a29b-740a13dcc42e\") " pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:54.054458 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054410    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-host\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.054458 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054448    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-host\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:54.054563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054474    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-kubelet-config\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:54.054563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054489    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-kubelet\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.054563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054515    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrzz\" (UniqueName: \"kubernetes.io/projected/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-api-access-ndrzz\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.054563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054531    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-config\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.054563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054545    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-daemon-config\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.054563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054559    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwkgq\" (UniqueName: \"kubernetes.io/projected/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-kube-api-access-lwkgq\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.054854 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054601    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:54.054854 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054637    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.054854 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054674    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.054854 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054713    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7pq5\" (UniqueName: \"kubernetes.io/projected/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-kube-api-access-j7pq5\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:54.054854 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054712    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-daemon-config\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.054854 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054757    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.054854 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054846    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-run\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.055073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054918    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-multus-certs\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.055073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054960    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-socket-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.055073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.054994    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kms8h\" (UniqueName: \"kubernetes.io/projected/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-kube-api-access-kms8h\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:54.055073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055021    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnkqf\" (UniqueName: \"kubernetes.io/projected/30affd33-d7b4-4d40-a944-a2568dde24a0-kube-api-access-vnkqf\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.055073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055038    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-bin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.055073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055069    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-hostroot\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.055312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055090    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lnrs\" (UniqueName: \"kubernetes.io/projected/9d18034b-20ee-4e14-9ae9-7c680ff195a1-kube-api-access-8lnrs\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:54.055312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055136    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-slash\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.055312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055207    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-node-log\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.055312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055225    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-var-lib-kubelet\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.055312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055252    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-tmp\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.055312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055268    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-netns\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.055312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055294    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-serviceca\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:54.055527 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055341    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgz84\" (UniqueName: \"kubernetes.io/projected/fd392127-738d-43fd-8e0a-f6e207feaf4a-kube-api-access-wgz84\") pod \"ingress-canary-d9llh\" (UID: \"fd392127-738d-43fd-8e0a-f6e207feaf4a\") " pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:54.055527 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055369    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4zj\" (UniqueName: \"kubernetes.io/projected/ffb9780a-2710-4742-b890-b7c0647a4086-kube-api-access-vn4zj\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:54.055527 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055472    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30affd33-d7b4-4d40-a944-a2568dde24a0-tmp\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.055527 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055499    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-var-lib-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.055655 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055529    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pcp\" (UniqueName: \"kubernetes.io/projected/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-kube-api-access-67pcp\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.055655 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055562    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-lib-modules\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.055655 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055584    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:54.055655 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055599    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cnibin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.055655 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055617    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-hosts-file\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:54.055655 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055650    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-cnibin\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.055888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055678    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-socket-dir-parent\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.055888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055697    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw57g\" (UniqueName: \"kubernetes.io/projected/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kube-api-access-jw57g\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.055888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055739    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-system-cni-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.055888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055841    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/126d815b-b83a-4f35-a8ac-38f52d1c9a10-data-volume\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.055888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055874    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-etc-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.056080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055899    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/866690aa-7dfb-4e2b-a29b-740a13dcc42e-agent-certs\") pod \"konnectivity-agent-sdtv7\" (UID: \"866690aa-7dfb-4e2b-a29b-740a13dcc42e\") " pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:54.056080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055911    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/126d815b-b83a-4f35-a8ac-38f52d1c9a10-data-volume\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.056080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055924    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-modprobe-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.056080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055966    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-kubernetes\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.056080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.055989    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.056080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.056022    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-device-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.056080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.056047    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.056080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.056069    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-systemd-units\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.056313 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.056244    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/866690aa-7dfb-4e2b-a29b-740a13dcc42e-agent-certs\") pod \"konnectivity-agent-sdtv7\" (UID: \"866690aa-7dfb-4e2b-a29b-740a13dcc42e\") " pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:54.103011 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.102918    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 19:15:54.103011 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.102942    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 19:15:54.103293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.103271    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovn-node-metrics-cert\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.103448 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.103433    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 19:15:54.103531 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.103506    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6brtl\""
Oct 30 19:15:54.103531 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.103506    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-csh58\""
Oct 30 19:15:54.103589 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.103562    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-qjhd5\""
Oct 30 19:15:54.103615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.103591    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd392127-738d-43fd-8e0a-f6e207feaf4a-cert\") pod \"ingress-canary-d9llh\" (UID: \"fd392127-738d-43fd-8e0a-f6e207feaf4a\") " pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:54.103810 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.103798    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 19:15:54.104111 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.104100    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 19:15:54.104203 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.104188    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jlfgq\""
Oct 30 19:15:54.104240 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.104220    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xzg6c\""
Oct 30 19:15:54.105759 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.105744    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n57d7\""
Oct 30 19:15:54.105851 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.105789    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gt8zl\""
Oct 30 19:15:54.106398 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106381    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4h2hd\""
Oct 30 19:15:54.106457 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106407    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hk6fn\""
Oct 30 19:15:54.106457 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106429    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q7svh\""
Oct 30 19:15:54.106457 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106438    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 19:15:54.106457 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106429    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnhjf\""
Oct 30 19:15:54.106457 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106450    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 19:15:54.106457 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106445    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106458    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106468    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-46fzl\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106531    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106636    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-px7h6\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106645    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106658    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106672    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106680    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106696    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rhxwm\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106707    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106699    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106707    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106738    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106744    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106746    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106735    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106742    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106723    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4k9jz\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106699    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106736    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106796    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106897    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106908    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106910    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106911    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106937    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106919    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106945    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.106989    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.109352    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5l6\" (UniqueName: \"kubernetes.io/projected/2b725985-0e36-4850-b8ae-d761df575eb8-kube-api-access-pv5l6\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.109494    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n76vm\" (UniqueName: \"kubernetes.io/projected/c4b32629-b4f3-4bb5-b8f2-ef5d0b634519-kube-api-access-n76vm\") pod \"network-check-target-qk5fg\" (UID: \"c4b32629-b4f3-4bb5-b8f2-ef5d0b634519\") " pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.109569    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fql4m\" (UniqueName: \"kubernetes.io/projected/d4008483-f3a6-4355-90df-db98415d8c2f-kube-api-access-fql4m\") pod \"migrator-66df5d496b-bv96d\" (UID: \"d4008483-f3a6-4355-90df-db98415d8c2f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bv96d"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.109602    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgzpv\" (UniqueName: \"kubernetes.io/projected/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-kube-api-access-zgzpv\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.113376    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.114503    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-env-overrides\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.114528    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb9780a-2710-4742-b890-b7c0647a4086-config-volume\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.114538    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-original-pull-secret\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.114604    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/866690aa-7dfb-4e2b-a29b-740a13dcc42e-konnectivity-ca\") pod \"konnectivity-agent-sdtv7\" (UID: \"866690aa-7dfb-4e2b-a29b-740a13dcc42e\") " pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.114625    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-script-lib\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.114715    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-ovnkube-config\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.115675    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2b725985-0e36-4850-b8ae-d761df575eb8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.115716    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-serviceca\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.116474    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnkqf\" (UniqueName: \"kubernetes.io/projected/30affd33-d7b4-4d40-a944-a2568dde24a0-kube-api-access-vnkqf\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.116924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.116493    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lnrs\" (UniqueName: \"kubernetes.io/projected/9d18034b-20ee-4e14-9ae9-7c680ff195a1-kube-api-access-8lnrs\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:54.152636 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.152609    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pcp\" (UniqueName: \"kubernetes.io/projected/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-kube-api-access-67pcp\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.152722 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.152701    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrzz\" (UniqueName: \"kubernetes.io/projected/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-api-access-ndrzz\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.153155 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.153125    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4zj\" (UniqueName: \"kubernetes.io/projected/ffb9780a-2710-4742-b890-b7c0647a4086-kube-api-access-vn4zj\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:54.153231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.153132    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwkgq\" (UniqueName: \"kubernetes.io/projected/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-kube-api-access-lwkgq\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.153231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.153184    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw57g\" (UniqueName: \"kubernetes.io/projected/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kube-api-access-jw57g\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.153335 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.153291    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgz84\" (UniqueName: \"kubernetes.io/projected/fd392127-738d-43fd-8e0a-f6e207feaf4a-kube-api-access-wgz84\") pod \"ingress-canary-d9llh\" (UID: \"fd392127-738d-43fd-8e0a-f6e207feaf4a\") " pod="openshift-ingress-canary/ingress-canary-d9llh"
Oct 30 19:15:54.153440 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.153423    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kms8h\" (UniqueName: \"kubernetes.io/projected/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-kube-api-access-kms8h\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:54.153645 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.153627    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7pq5\" (UniqueName: \"kubernetes.io/projected/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-kube-api-access-j7pq5\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:54.156946 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.156931    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:54.156987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.156963    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cnibin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.156987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.156980    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-hosts-file\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:54.157048 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.156994    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-cnibin\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.157048 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157008    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-socket-dir-parent\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.157048 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157012    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-cnibin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.157048 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.157022    4420 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:15:54.157048 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157042    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-system-cni-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157066    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2d3697f-f464-4a7d-af17-4ea6d73ae5ac-hosts-file\") pod \"node-resolver-x2pz6\" (UID: \"f2d3697f-f464-4a7d-af17-4ea6d73ae5ac\") " pod="openshift-dns/node-resolver-x2pz6"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157081    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-system-cni-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157088    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-cnibin\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157099    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-socket-dir-parent\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.157106    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.657089055 +0000 UTC m=+1.868306200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : secret "metrics-daemon-secret" not found
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157126    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-etc-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157145    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-modprobe-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157158    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-kubernetes\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157160    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-etc-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157171    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157186    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-device-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157193    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-kubernetes\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.157202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157205    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157219    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-systemd-units\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157222    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-device-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157238    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-modprobe-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157245    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-systemd-units\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157260    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157284    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157294    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-d\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157262    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157318    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-ovn\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157299    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-ovn\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157351    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-log-socket\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157366    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157376    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-systemd\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157404    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-log-socket\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157402    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.157413    4420 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157431    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-sys-fs\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157435    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.157439    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls podName:126d815b-b83a-4f35-a8ac-38f52d1c9a10 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.657430888 +0000 UTC m=+1.868648011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqj9m" (UID: "126d815b-b83a-4f35-a8ac-38f52d1c9a10") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157462    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-sys-fs\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157486    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-systemd\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157490    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-sys\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157510    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-registration-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157526    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d18034b-20ee-4e14-9ae9-7c680ff195a1-host-slash\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157553    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d18034b-20ee-4e14-9ae9-7c680ff195a1-host-slash\") pod \"iptables-alerter-dx9vb\" (UID: \"9d18034b-20ee-4e14-9ae9-7c680ff195a1\") " pod="openshift-network-operator/iptables-alerter-dx9vb"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157560    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-bin\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157558    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-sys\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157568    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-registration-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157599    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-bin\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157615    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157629    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-netns\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157643    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-systemd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.157641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157656    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157680    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysconfig\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157696    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-kubelet\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157703    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-systemd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157715    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-run-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157719    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-etc-kubernetes\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157727    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-run-netns\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157742    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-kubelet\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157751    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-conf\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157755    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysconfig\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157756    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-etc-kubernetes\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157774    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-system-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157795    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-conf-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157831    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-dbus\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157854    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-system-cni-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157854    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/126d815b-b83a-4f35-a8ac-38f52d1c9a10-crio-socket\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157874    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-etc-sysctl-conf\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157880    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-k8s-cni-cncf-io\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157901    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-dbus\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157906    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-multus\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157865    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-multus-conf-dir\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157922    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-k8s-cni-cncf-io\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157926    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/126d815b-b83a-4f35-a8ac-38f52d1c9a10-crio-socket\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157958    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-netd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157965    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-multus\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.157983    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158013    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158030    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-cni-netd\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158064    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-host\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158088    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-host\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158117    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-kubelet-config\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158122    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/126d815b-b83a-4f35-a8ac-38f52d1c9a10-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158140    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/093a7b2a-fb5b-4703-8073-98c60f1ef5ca-host\") pod \"node-ca-hlnt2\" (UID: \"093a7b2a-fb5b-4703-8073-98c60f1ef5ca\") " pod="openshift-image-registry/node-ca-hlnt2"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158142    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-kubelet\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158142    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-host\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158163    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158173    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51-kubelet-config\") pod \"global-pull-secret-syncer-n5gpj\" (UID: \"d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51\") " pod="kube-system/global-pull-secret-syncer-n5gpj"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158175    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-kubelet\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158179    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158207    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2b725985-0e36-4850-b8ae-d761df575eb8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-475ns\" (UID: \"2b725985-0e36-4850-b8ae-d761df575eb8\") " pod="openshift-multus/multus-additional-cni-plugins-475ns"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.158218    4420 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158219    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158244    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-run\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.158251    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls podName:ffb9780a-2710-4742-b890-b7c0647a4086 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:54.658242398 +0000 UTC m=+1.869459521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls") pod "dns-default-2gdtt" (UID: "ffb9780a-2710-4742-b890-b7c0647a4086") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158265    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-multus-certs\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158246    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158281    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-socket-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158280    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-run\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158303    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-bin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158314    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-multus-certs\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158318    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-hostroot\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158323    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/98a8605a-fd55-4808-b74c-cd77b35fdbd3-socket-dir\") pod \"aws-ebs-csi-driver-node-hq4x8\" (UID: \"98a8605a-fd55-4808-b74c-cd77b35fdbd3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hq4x8"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158336    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-slash\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158342    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-hostroot\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158342    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-var-lib-cni-bin\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158352    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-node-log\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158364    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-host-slash\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158368    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-var-lib-kubelet\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158386    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-netns\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158387    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-node-log\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158395    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-var-lib-kubelet\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158403    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-var-lib-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158417    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75c29e8f-e445-40b1-a2c0-1e8252b8f53f-host-run-netns\") pod \"multus-hr8dq\" (UID: \"75c29e8f-e445-40b1-a2c0-1e8252b8f53f\") " pod="openshift-multus/multus-hr8dq"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158417    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-lib-modules\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158456    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be-var-lib-openvswitch\") pod \"ovnkube-node-vspvz\" (UID: \"c1bd44c9-6ec3-4eaf-82fc-0d8f7c1264be\") " pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:54.158713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.158480    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30affd33-d7b4-4d40-a944-a2568dde24a0-lib-modules\") pod \"tuned-n7llw\" (UID: \"30affd33-d7b4-4d40-a944-a2568dde24a0\") " pod="openshift-cluster-node-tuning-operator/tuned-n7llw"
Oct 30 19:15:54.662572 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.662544    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:54.662701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.662583    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:54.662701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:54.662606    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:54.662778 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.662704    4420 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:54.662778 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.662708    4420 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:15:54.662778 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.662708    4420 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:54.662778 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.662753    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls podName:126d815b-b83a-4f35-a8ac-38f52d1c9a10 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.662741515 +0000 UTC m=+2.873958638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqj9m" (UID: "126d815b-b83a-4f35-a8ac-38f52d1c9a10") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:54.662911 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.662784    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.662773446 +0000 UTC m=+2.873990569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : secret "metrics-daemon-secret" not found
Oct 30 19:15:54.662911 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:54.662794    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls podName:ffb9780a-2710-4742-b890-b7c0647a4086 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:55.662789174 +0000 UTC m=+2.874006297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls") pod "dns-default-2gdtt" (UID: "ffb9780a-2710-4742-b890-b7c0647a4086") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:55.669048 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:55.669019    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:55.669371 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:55.669057    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:55.669371 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:55.669081    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:55.669371 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:55.669164    4420 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:55.669371 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:55.669165    4420 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:55.669371 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:55.669203    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls podName:126d815b-b83a-4f35-a8ac-38f52d1c9a10 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.669192405 +0000 UTC m=+4.880409529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqj9m" (UID: "126d815b-b83a-4f35-a8ac-38f52d1c9a10") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:55.669371 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:55.669226    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls podName:ffb9780a-2710-4742-b890-b7c0647a4086 nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.669214061 +0000 UTC m=+4.880431185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls") pod "dns-default-2gdtt" (UID: "ffb9780a-2710-4742-b890-b7c0647a4086") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:55.669371 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:55.669165    4420 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:15:55.669371 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:55.669252    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:15:57.669245411 +0000 UTC m=+4.880462534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : secret "metrics-daemon-secret" not found
Oct 30 19:15:56.398406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:56.398377    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:57.679784 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:57.679741    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:15:57.680311 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:57.679805    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:15:57.680311 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:57.679864    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:15:57.680311 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:57.679906    4420 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:15:57.680311 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:57.679960    4420 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:57.680311 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:57.679966    4420 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 19:15:57.680311 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:57.679974    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.679960099 +0000 UTC m=+8.891177242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : secret "metrics-daemon-secret" not found
Oct 30 19:15:57.680311 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:57.680038    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls podName:126d815b-b83a-4f35-a8ac-38f52d1c9a10 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.680022982 +0000 UTC m=+8.891240111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqj9m" (UID: "126d815b-b83a-4f35-a8ac-38f52d1c9a10") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:15:57.680311 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:15:57.680053    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls podName:ffb9780a-2710-4742-b890-b7c0647a4086 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:01.680043316 +0000 UTC m=+8.891260440 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls") pod "dns-default-2gdtt" (UID: "ffb9780a-2710-4742-b890-b7c0647a4086") : secret "dns-default-metrics-tls" not found
Oct 30 19:15:58.224001 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:58.223971    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:15:58.224190 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:58.224176    4420 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 19:15:59.986006 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:15:59.985980    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x2pz6_f2d3697f-f464-4a7d-af17-4ea6d73ae5ac/dns-node-resolver/0.log"
Oct 30 19:16:00.585603 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:00.585574    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hlnt2_093a7b2a-fb5b-4703-8073-98c60f1ef5ca/node-ca/0.log"
Oct 30 19:16:01.584973 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:01.584949    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d9llh_fd392127-738d-43fd-8e0a-f6e207feaf4a/serve-healthcheck-canary/0.log"
Oct 30 19:16:01.705477 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:01.705456    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:16:01.705575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:01.705493    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:16:01.705575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:01.705525    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:16:01.705674 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:16:01.705626    4420 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 19:16:01.705674 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:16:01.705627    4420 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:01.705762 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:16:01.705712    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls podName:126d815b-b83a-4f35-a8ac-38f52d1c9a10 nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.705688434 +0000 UTC m=+16.916905571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqj9m" (UID: "126d815b-b83a-4f35-a8ac-38f52d1c9a10") : secret "insights-runtime-extractor-tls" not found
Oct 30 19:16:01.705762 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:16:01.705730    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs podName:c36c31f3-8f2c-4201-b342-9d5c5bab7b6f nodeName:}" failed. No retries permitted until 2026-10-30 19:16:09.705720317 +0000 UTC m=+16.916937457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs") pod "network-metrics-daemon-gwzbc" (UID: "c36c31f3-8f2c-4201-b342-9d5c5bab7b6f") : secret "metrics-daemon-secret" not found
Oct 30 19:16:01.705902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:01.705877    4420 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 19:16:01.707963 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:01.707948    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffb9780a-2710-4742-b890-b7c0647a4086-metrics-tls\") pod \"dns-default-2gdtt\" (UID: \"ffb9780a-2710-4742-b890-b7c0647a4086\") " pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:16:01.754083 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:01.754063    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:16:01.985290 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:01.985261    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-bv96d_d4008483-f3a6-4355-90df-db98415d8c2f/migrator/0.log"
Oct 30 19:16:02.119017 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:02.118945    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2gdtt"]
Oct 30 19:16:02.124905 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:02.124854    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb9780a_2710_4742_b890_b7c0647a4086.slice/crio-d6a7a2016b14a93b6e10ab141920e5dd932d732e5a76752b120afe4a79f652d5 WatchSource:0}: Error finding container d6a7a2016b14a93b6e10ab141920e5dd932d732e5a76752b120afe4a79f652d5: Status 404 returned error can't find the container with id d6a7a2016b14a93b6e10ab141920e5dd932d732e5a76752b120afe4a79f652d5
Oct 30 19:16:02.127161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:02.127146    4420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:16:02.248873 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:02.248796    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vspvz"
Oct 30 19:16:02.944547 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:02.944517    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2gdtt" event={"ID":"ffb9780a-2710-4742-b890-b7c0647a4086","Type":"ContainerStarted","Data":"d6a7a2016b14a93b6e10ab141920e5dd932d732e5a76752b120afe4a79f652d5"}
Oct 30 19:16:03.523782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:03.523752    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:16:03.595053 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:03.595024    4420 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-7.us-west-1.compute.internal" event="NodeReady"
Oct 30 19:16:04.951374 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:04.951345    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2gdtt" event={"ID":"ffb9780a-2710-4742-b890-b7c0647a4086","Type":"ContainerStarted","Data":"fa135b762716b6f3abc50f00c54cae5c2a23a04d3d185f3cebedba84a4ddf0a1"}
Oct 30 19:16:04.951720 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:04.951389    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2gdtt" event={"ID":"ffb9780a-2710-4742-b890-b7c0647a4086","Type":"ContainerStarted","Data":"81db8f65291987852c3406a3051ce11dd0a35de519433e0770ce48aaecb90a61"}
Oct 30 19:16:04.951720 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:04.951479    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:16:09.760689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:09.760659    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:16:09.761089 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:09.760698    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:16:09.762940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:09.762916    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c36c31f3-8f2c-4201-b342-9d5c5bab7b6f-metrics-certs\") pod \"network-metrics-daemon-gwzbc\" (UID: \"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f\") " pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:16:09.763051 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:09.762994    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/126d815b-b83a-4f35-a8ac-38f52d1c9a10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqj9m\" (UID: \"126d815b-b83a-4f35-a8ac-38f52d1c9a10\") " pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:16:09.850757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:09.850733    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwzbc"
Oct 30 19:16:09.856624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:09.856603    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tqj9m"
Oct 30 19:16:10.220069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:10.220043    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gwzbc"]
Oct 30 19:16:10.259121 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:10.259099    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36c31f3_8f2c_4201_b342_9d5c5bab7b6f.slice/crio-c16abb230aded14ac1ed1490a41b842dc886589e2dc7ecc2bffc0f91014a3dca WatchSource:0}: Error finding container c16abb230aded14ac1ed1490a41b842dc886589e2dc7ecc2bffc0f91014a3dca: Status 404 returned error can't find the container with id c16abb230aded14ac1ed1490a41b842dc886589e2dc7ecc2bffc0f91014a3dca
Oct 30 19:16:10.269469 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:10.269450    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tqj9m"]
Oct 30 19:16:10.272365 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:10.272348    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126d815b_b83a_4f35_a8ac_38f52d1c9a10.slice/crio-3ae6ac904b84d481f56d9e80357518a630eb6c4ee0872520a3283c1d3c7386f1 WatchSource:0}: Error finding container 3ae6ac904b84d481f56d9e80357518a630eb6c4ee0872520a3283c1d3c7386f1: Status 404 returned error can't find the container with id 3ae6ac904b84d481f56d9e80357518a630eb6c4ee0872520a3283c1d3c7386f1
Oct 30 19:16:10.967511 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:10.967480    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqj9m" event={"ID":"126d815b-b83a-4f35-a8ac-38f52d1c9a10","Type":"ContainerStarted","Data":"fc686a5511673d02a8488575fa8b08a29622c505e3b9461422bd141ce4b7d37a"}
Oct 30 19:16:10.967924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:10.967531    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqj9m" event={"ID":"126d815b-b83a-4f35-a8ac-38f52d1c9a10","Type":"ContainerStarted","Data":"3ae6ac904b84d481f56d9e80357518a630eb6c4ee0872520a3283c1d3c7386f1"}
Oct 30 19:16:10.968635 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:10.968613    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gwzbc" event={"ID":"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f","Type":"ContainerStarted","Data":"c16abb230aded14ac1ed1490a41b842dc886589e2dc7ecc2bffc0f91014a3dca"}
Oct 30 19:16:11.977919 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:11.977840    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqj9m" event={"ID":"126d815b-b83a-4f35-a8ac-38f52d1c9a10","Type":"ContainerStarted","Data":"57668d9f67e3c9bcf2a5ee76514dd3de742fbd7edba1f1d82b6a7e83e865a09a"}
Oct 30 19:16:11.979537 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:11.979513    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gwzbc" event={"ID":"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f","Type":"ContainerStarted","Data":"c0f64dcc480246d2dc76fd88aa4ca42c2bf3ae597d56a3ec733a6f8705da24b3"}
Oct 30 19:16:11.979651 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:11.979562    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gwzbc" event={"ID":"c36c31f3-8f2c-4201-b342-9d5c5bab7b6f","Type":"ContainerStarted","Data":"723d6346d199023606a9d95ab409402c5a56df0f8453ad34d0bd3643ec0d4af0"}
Oct 30 19:16:13.988145 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:13.988114    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqj9m" event={"ID":"126d815b-b83a-4f35-a8ac-38f52d1c9a10","Type":"ContainerStarted","Data":"3bfa6e95d73fa26ca3841d5d3ac55b9da548ceaa019e73c1f56c1fb856fbd224"}
Oct 30 19:16:14.955643 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:14.955613    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2gdtt"
Oct 30 19:16:18.466654 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.466624    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"]
Oct 30 19:16:18.469065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.469049    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.510176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.510151    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c199259-8a97-4dfe-a229-259283ca4494-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.510291 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.510184    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvchg\" (UniqueName: \"kubernetes.io/projected/3c199259-8a97-4dfe-a229-259283ca4494-kube-api-access-bvchg\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.510291 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.510211    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c199259-8a97-4dfe-a229-259283ca4494-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.510379 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.510296    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c199259-8a97-4dfe-a229-259283ca4494-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.570640 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.570616    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 19:16:18.570930 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.570918    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 19:16:18.570973 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.570926    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 19:16:18.570973 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.570946    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 19:16:18.571093 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.571079    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 19:16:18.610913 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.610890    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c199259-8a97-4dfe-a229-259283ca4494-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.611007 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.610923    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c199259-8a97-4dfe-a229-259283ca4494-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.611007 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.610941    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvchg\" (UniqueName: \"kubernetes.io/projected/3c199259-8a97-4dfe-a229-259283ca4494-kube-api-access-bvchg\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.611007 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.610964    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c199259-8a97-4dfe-a229-259283ca4494-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.611528 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.611499    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c199259-8a97-4dfe-a229-259283ca4494-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.613304 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.613285    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c199259-8a97-4dfe-a229-259283ca4494-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.613495 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.613476    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c199259-8a97-4dfe-a229-259283ca4494-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.620124 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.620108    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zm528\""
Oct 30 19:16:18.621095 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.621077    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"]
Oct 30 19:16:18.668178 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.668157    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvchg\" (UniqueName: \"kubernetes.io/projected/3c199259-8a97-4dfe-a229-259283ca4494-kube-api-access-bvchg\") pod \"prometheus-operator-9cd89b74c-knpkc\" (UID: \"3c199259-8a97-4dfe-a229-259283ca4494\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:18.779306 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:18.779232    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"
Oct 30 19:16:19.145120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:19.145090    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-knpkc"]
Oct 30 19:16:19.148652 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:19.148625    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c199259_8a97_4dfe_a229_259283ca4494.slice/crio-1c0e96907bb280677b007c2ec0f7801a938e55e5e27993c004262d2bacc45947 WatchSource:0}: Error finding container 1c0e96907bb280677b007c2ec0f7801a938e55e5e27993c004262d2bacc45947: Status 404 returned error can't find the container with id 1c0e96907bb280677b007c2ec0f7801a938e55e5e27993c004262d2bacc45947
Oct 30 19:16:20.005470 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:20.005427    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc" event={"ID":"3c199259-8a97-4dfe-a229-259283ca4494","Type":"ContainerStarted","Data":"1c0e96907bb280677b007c2ec0f7801a938e55e5e27993c004262d2bacc45947"}
Oct 30 19:16:21.010064 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:21.010026    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc" event={"ID":"3c199259-8a97-4dfe-a229-259283ca4494","Type":"ContainerStarted","Data":"82db07be5223c3c89abd10a8063d5b181f7be62e93c762a85aed885554a4f819"}
Oct 30 19:16:21.010404 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:21.010092    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc" event={"ID":"3c199259-8a97-4dfe-a229-259283ca4494","Type":"ContainerStarted","Data":"6a834e8db03e78a140d1f5358c46b62f40a4f563f60e0b2e4855ea7984938dcd"}
Oct 30 19:16:21.172244 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:21.172183    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-knpkc" podStartSLOduration=1.543342039 podStartE2EDuration="3.172166546s" totalImagesPullingTime="1.628824507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:21.171248326 +0000 UTC m=+28.382465481" watchObservedRunningTime="2026-10-30 19:16:21.172166546 +0000 UTC m=+28.383383692"
Oct 30 19:16:23.074168 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.074126    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sdtv7"
Oct 30 19:16:23.526667 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.526636    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qk5fg"
Oct 30 19:16:23.811099 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.810999    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"]
Oct 30 19:16:23.844349 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.844327    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:23.874996 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.874899    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"]
Oct 30 19:16:23.900638 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.900614    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-72ppw\""
Oct 30 19:16:23.900638 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.900628    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:16:23.923404 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.923387    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 19:16:23.947054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.947030    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3c96fe5-1b6c-4385-a076-2705589f19f6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:23.947139 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.947065    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3c96fe5-1b6c-4385-a076-2705589f19f6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:23.947139 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.947098    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3c96fe5-1b6c-4385-a076-2705589f19f6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:23.947220 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.947166    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52bzg\" (UniqueName: \"kubernetes.io/projected/d3c96fe5-1b6c-4385-a076-2705589f19f6-kube-api-access-52bzg\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:23.949592 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.949572    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"]
Oct 30 19:16:23.964144 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.964129    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:23.998392 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:23.998373    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cfzh5"]
Oct 30 19:16:24.010963 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.010944    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.012477 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.012455    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"]
Oct 30 19:16:24.017553 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.017533    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 19:16:24.020603 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.018387    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 19:16:24.020603 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.018461    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 19:16:24.020603 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.018562    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9zvn6\""
Oct 30 19:16:24.048173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048152    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.048265 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048183    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.048265 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048207    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52bzg\" (UniqueName: \"kubernetes.io/projected/d3c96fe5-1b6c-4385-a076-2705589f19f6-kube-api-access-52bzg\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:24.048265 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048222    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0973102-c49b-4ef1-91d4-29357c38bd96-root\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.048385 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048337    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3c96fe5-1b6c-4385-a076-2705589f19f6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:24.048385 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048378    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvvn\" (UniqueName: \"kubernetes.io/projected/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-api-access-bdvvn\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.048457 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048397    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3c96fe5-1b6c-4385-a076-2705589f19f6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:24.048457 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048429    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.048457 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048447    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt9mf\" (UniqueName: \"kubernetes.io/projected/d0973102-c49b-4ef1-91d4-29357c38bd96-kube-api-access-zt9mf\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.048632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048466    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4bbc31a-54e5-447c-b2fa-db1f16caa108-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.048632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048524    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.048733 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048650    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0973102-c49b-4ef1-91d4-29357c38bd96-sys\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.048846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048728    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3c96fe5-1b6c-4385-a076-2705589f19f6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:24.048846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048763    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-wtmp\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.048846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048789    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-tls\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.048954 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048842    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c4bbc31a-54e5-447c-b2fa-db1f16caa108-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.048954 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048921    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.049028 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048953    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-textfile\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.049028 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048975    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3c96fe5-1b6c-4385-a076-2705589f19f6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:24.049028 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.048975    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0973102-c49b-4ef1-91d4-29357c38bd96-metrics-client-ca\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.050796 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.050775    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3c96fe5-1b6c-4385-a076-2705589f19f6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:24.051198 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.051171    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3c96fe5-1b6c-4385-a076-2705589f19f6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:24.072528 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.072473    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 19:16:24.072787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.072774    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2dwz4\""
Oct 30 19:16:24.072787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.072783    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 19:16:24.105598 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.105573    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52bzg\" (UniqueName: \"kubernetes.io/projected/d3c96fe5-1b6c-4385-a076-2705589f19f6-kube-api-access-52bzg\") pod \"openshift-state-metrics-7bd7489957-bwnjq\" (UID: \"d3c96fe5-1b6c-4385-a076-2705589f19f6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:24.117469 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.117452    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 19:16:24.150139 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150122    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4bbc31a-54e5-447c-b2fa-db1f16caa108-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.150226 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150147    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.150226 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150169    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0973102-c49b-4ef1-91d4-29357c38bd96-sys\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150226 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150187    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-wtmp\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150226 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150208    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-tls\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150236    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c4bbc31a-54e5-447c-b2fa-db1f16caa108-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.150418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150272    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d0973102-c49b-4ef1-91d4-29357c38bd96-sys\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150310    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.150418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150332    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-textfile\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150355    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0973102-c49b-4ef1-91d4-29357c38bd96-metrics-client-ca\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150381    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150389    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-wtmp\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150411    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.150805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150445    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0973102-c49b-4ef1-91d4-29357c38bd96-root\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150489    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvvn\" (UniqueName: \"kubernetes.io/projected/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-api-access-bdvvn\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.150805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150517    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150544    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt9mf\" (UniqueName: \"kubernetes.io/projected/d0973102-c49b-4ef1-91d4-29357c38bd96-kube-api-access-zt9mf\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150698    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-textfile\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150768    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d0973102-c49b-4ef1-91d4-29357c38bd96-root\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.150805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150755    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c4bbc31a-54e5-447c-b2fa-db1f16caa108-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.151163 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.150950    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4bbc31a-54e5-447c-b2fa-db1f16caa108-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.151163 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.151050    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0973102-c49b-4ef1-91d4-29357c38bd96-metrics-client-ca\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.151163 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.151054    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.151313 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.151214    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-accelerators-collector-config\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.152654 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.152631    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.152963 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.152945    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.153255 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.153239    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.153523 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.153501    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d0973102-c49b-4ef1-91d4-29357c38bd96-node-exporter-tls\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.193893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.193878    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"
Oct 30 19:16:24.208556 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.208533    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvvn\" (UniqueName: \"kubernetes.io/projected/c4bbc31a-54e5-447c-b2fa-db1f16caa108-kube-api-access-bdvvn\") pod \"kube-state-metrics-5dc8d7d89b-d27dr\" (UID: \"c4bbc31a-54e5-447c-b2fa-db1f16caa108\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.208638 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.208536    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt9mf\" (UniqueName: \"kubernetes.io/projected/d0973102-c49b-4ef1-91d4-29357c38bd96-kube-api-access-zt9mf\") pod \"node-exporter-cfzh5\" (UID: \"d0973102-c49b-4ef1-91d4-29357c38bd96\") " pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.312155 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.312126    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"
Oct 30 19:16:24.390624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.390595    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cfzh5"
Oct 30 19:16:24.421242 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:24.421217    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0973102_c49b_4ef1_91d4_29357c38bd96.slice/crio-c5791abc992cddd829c12273bbfaa0b16b5c01e6caf482cc2215ba43bdf28b4b WatchSource:0}: Error finding container c5791abc992cddd829c12273bbfaa0b16b5c01e6caf482cc2215ba43bdf28b4b: Status 404 returned error can't find the container with id c5791abc992cddd829c12273bbfaa0b16b5c01e6caf482cc2215ba43bdf28b4b
Oct 30 19:16:24.657288 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.657225    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq"]
Oct 30 19:16:24.659507 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:24.659480    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c96fe5_1b6c_4385_a076_2705589f19f6.slice/crio-b9a4ccf268dec6e4c3b9e9f1e1215b6543fe35070fd126cf043ee0b0c56559a2 WatchSource:0}: Error finding container b9a4ccf268dec6e4c3b9e9f1e1215b6543fe35070fd126cf043ee0b0c56559a2: Status 404 returned error can't find the container with id b9a4ccf268dec6e4c3b9e9f1e1215b6543fe35070fd126cf043ee0b0c56559a2
Oct 30 19:16:24.719976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:24.719954    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr"]
Oct 30 19:16:24.722806 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:24.722785    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bbc31a_54e5_447c_b2fa_db1f16caa108.slice/crio-e6ad237edafbaf13cdd6f4afc380b88878f456a0a75460c3310066e1fc9f4c46 WatchSource:0}: Error finding container e6ad237edafbaf13cdd6f4afc380b88878f456a0a75460c3310066e1fc9f4c46: Status 404 returned error can't find the container with id e6ad237edafbaf13cdd6f4afc380b88878f456a0a75460c3310066e1fc9f4c46
Oct 30 19:16:25.023033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:25.022956    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr" event={"ID":"c4bbc31a-54e5-447c-b2fa-db1f16caa108","Type":"ContainerStarted","Data":"e6ad237edafbaf13cdd6f4afc380b88878f456a0a75460c3310066e1fc9f4c46"}
Oct 30 19:16:25.024141 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:25.024117    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfzh5" event={"ID":"d0973102-c49b-4ef1-91d4-29357c38bd96","Type":"ContainerStarted","Data":"c5791abc992cddd829c12273bbfaa0b16b5c01e6caf482cc2215ba43bdf28b4b"}
Oct 30 19:16:25.025833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:25.025791    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq" event={"ID":"d3c96fe5-1b6c-4385-a076-2705589f19f6","Type":"ContainerStarted","Data":"4ac9b8f85bbc4c182f7e466719507e8caae9a463dcfc8c6852697f2c27b8f891"}
Oct 30 19:16:25.025940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:25.025860    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq" event={"ID":"d3c96fe5-1b6c-4385-a076-2705589f19f6","Type":"ContainerStarted","Data":"8455273886d0dc8b64aa2c6545e55166f77bd0340aa3e70a118a021d7f80873b"}
Oct 30 19:16:25.025940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:25.025889    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq" event={"ID":"d3c96fe5-1b6c-4385-a076-2705589f19f6","Type":"ContainerStarted","Data":"b9a4ccf268dec6e4c3b9e9f1e1215b6543fe35070fd126cf043ee0b0c56559a2"}
Oct 30 19:16:26.222602 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.222566    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:16:26.256433 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.256396    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.308943 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.308912    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:16:26.308943 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.308931    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:16:26.308943 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.308942    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:16:26.309185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.308951    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:16:26.309185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.308974    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:16:26.309185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.308983    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:16:26.309185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.309016    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:16:26.309185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.309020    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d724h\""
Oct 30 19:16:26.321627 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.321606    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:16:26.363949 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.363912    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364095 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.363965    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364095 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.363989    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364095 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.364027    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs8hv\" (UniqueName: \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-kube-api-access-rs8hv\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364095 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.364045    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364095 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.364059    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364095 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.364086    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364372 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.364112    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-volume\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364372 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.364142    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-web-config\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364372 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.364171    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-out\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364372 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.364203    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364372 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.364224    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.364372 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.364257    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.408152 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.408124    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:16:26.409852 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.409797    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:16:26.465454 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.465427    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.465577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.465461    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-volume\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.465577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.465494    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-web-config\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.465644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.465621    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-out\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.465712 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.465694    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.465755 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.465727    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.465800 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.465758    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.466575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.465912    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.466575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.465959    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.466575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.465983    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.466575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.466035    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs8hv\" (UniqueName: \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-kube-api-access-rs8hv\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.466575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.466055    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.466575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.466065    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.466575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.466090    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.467048 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.466637    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.467048 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.466890    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.468642 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.468586    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-out\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.468825 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.468791    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-web-config\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.469415 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.469380    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.469534 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.469470    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.469619 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.469603    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.469879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.469802    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.470011 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.469990    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.470079 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.470034    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-volume\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.470667 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.470646    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.523511 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.523456    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs8hv\" (UniqueName: \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-kube-api-access-rs8hv\") pod \"alertmanager-main-0\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:26.676867 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:26.676838    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:16:27.038709 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:27.038682    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq" event={"ID":"d3c96fe5-1b6c-4385-a076-2705589f19f6","Type":"ContainerStarted","Data":"d9a897238226d0545dd411db6cf6fb33d2e5fdcfc38c430482f32402a4297a79"}
Oct 30 19:16:27.411467 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:27.411443    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:16:27.464632 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:27.464608    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14d5a8d_d60b_4c39_b6a7_3a3b9f2f05ca.slice/crio-3463e58f1dec6efa8e41da415a14272771351b5ed015341880eb697b6e20d481 WatchSource:0}: Error finding container 3463e58f1dec6efa8e41da415a14272771351b5ed015341880eb697b6e20d481: Status 404 returned error can't find the container with id 3463e58f1dec6efa8e41da415a14272771351b5ed015341880eb697b6e20d481
Oct 30 19:16:28.043030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.043001    4420 generic.go:356] "Generic (PLEG): container finished" podID="d0973102-c49b-4ef1-91d4-29357c38bd96" containerID="ee8eba06d24f65a14022f2064fa416d77a7eae64aa6bb54f2f7ab702565b8b2c" exitCode=0
Oct 30 19:16:28.043224 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.043077    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfzh5" event={"ID":"d0973102-c49b-4ef1-91d4-29357c38bd96","Type":"ContainerDied","Data":"ee8eba06d24f65a14022f2064fa416d77a7eae64aa6bb54f2f7ab702565b8b2c"}
Oct 30 19:16:28.046192 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.046052    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerStarted","Data":"3463e58f1dec6efa8e41da415a14272771351b5ed015341880eb697b6e20d481"}
Oct 30 19:16:28.048747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.048611    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr" event={"ID":"c4bbc31a-54e5-447c-b2fa-db1f16caa108","Type":"ContainerStarted","Data":"a7be490a5cc5535274522af5ddf1bf601798f74f22f2acc589a57f8ee5e7d584"}
Oct 30 19:16:28.048747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.048663    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr" event={"ID":"c4bbc31a-54e5-447c-b2fa-db1f16caa108","Type":"ContainerStarted","Data":"57e5511be838bfee654bdb156c2e37ffcffa9c64c87a78fe6fcf8127a7c372db"}
Oct 30 19:16:28.048747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.048691    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr" event={"ID":"c4bbc31a-54e5-447c-b2fa-db1f16caa108","Type":"ContainerStarted","Data":"afebc04d615aa40854e9ddcbd7c0ad5260dec1920f4fbd5d82216810cee7c8d9"}
Oct 30 19:16:28.419609 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.419554    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-d27dr" podStartSLOduration=2.979085602 podStartE2EDuration="5.419536925s" totalImagesPullingTime="2.440451323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:28.291793143 +0000 UTC m=+35.503010290" watchObservedRunningTime="2026-10-30 19:16:28.419536925 +0000 UTC m=+35.630754073"
Oct 30 19:16:28.420267 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.420236    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-bwnjq" podStartSLOduration=3.3344408469999998 podStartE2EDuration="5.42022519s" totalImagesPullingTime="2.085784343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:28.418289927 +0000 UTC m=+35.629507072" watchObservedRunningTime="2026-10-30 19:16:28.42022519 +0000 UTC m=+35.631442358"
Oct 30 19:16:28.927366 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.927336    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f99b99bd4-fqbcv"]
Oct 30 19:16:28.929582 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.929560    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:28.982785 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.982758    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 19:16:28.983070 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.983052    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 19:16:28.983070 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.983069    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-twvzw\""
Oct 30 19:16:28.983185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.983062    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 19:16:28.983185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.983093    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 19:16:28.983185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.983148    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bqfior14m2ki0\""
Oct 30 19:16:28.984380 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.984362    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6f417180-aa17-47d5-9d7e-0672706c1847-secret-metrics-server-client-certs\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:28.984472 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.984395    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f417180-aa17-47d5-9d7e-0672706c1847-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:28.984472 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.984432    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6f417180-aa17-47d5-9d7e-0672706c1847-secret-metrics-server-tls\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:28.984587 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.984484    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6f417180-aa17-47d5-9d7e-0672706c1847-audit-log\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:28.984587 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.984551    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6f417180-aa17-47d5-9d7e-0672706c1847-metrics-server-audit-profiles\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:28.984689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.984642    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6f417180-aa17-47d5-9d7e-0672706c1847-client-ca-bundle\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:28.984689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.984675    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjb6l\" (UniqueName: \"kubernetes.io/projected/6f417180-aa17-47d5-9d7e-0672706c1847-kube-api-access-gjb6l\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:28.989287 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:28.989267    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f99b99bd4-fqbcv"]
Oct 30 19:16:29.055161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.055131    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfzh5" event={"ID":"d0973102-c49b-4ef1-91d4-29357c38bd96","Type":"ContainerStarted","Data":"9c363f44b142abfacb55015b1cc6dc8f87fce44b504c1a3e50c0a74e4153e1d9"}
Oct 30 19:16:29.055290 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.055187    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cfzh5" event={"ID":"d0973102-c49b-4ef1-91d4-29357c38bd96","Type":"ContainerStarted","Data":"dfdad70102e9d54127262061a7ce1ab1ed4a4d5536925e8bcb2e218a1208bcc2"}
Oct 30 19:16:29.085605 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.085582    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6f417180-aa17-47d5-9d7e-0672706c1847-client-ca-bundle\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.085741 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.085618    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjb6l\" (UniqueName: \"kubernetes.io/projected/6f417180-aa17-47d5-9d7e-0672706c1847-kube-api-access-gjb6l\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.085847 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.085826    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6f417180-aa17-47d5-9d7e-0672706c1847-secret-metrics-server-client-certs\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.085922 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.085888    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f417180-aa17-47d5-9d7e-0672706c1847-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.085976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.085935    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6f417180-aa17-47d5-9d7e-0672706c1847-secret-metrics-server-tls\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.085976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.085963    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6f417180-aa17-47d5-9d7e-0672706c1847-audit-log\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.086077 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.085991    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6f417180-aa17-47d5-9d7e-0672706c1847-metrics-server-audit-profiles\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.086555 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.086445    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6f417180-aa17-47d5-9d7e-0672706c1847-audit-log\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.086912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.086887    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f417180-aa17-47d5-9d7e-0672706c1847-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.087139 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.087121    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6f417180-aa17-47d5-9d7e-0672706c1847-metrics-server-audit-profiles\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.088468 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.088448    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6f417180-aa17-47d5-9d7e-0672706c1847-client-ca-bundle\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.089070 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.089044    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6f417180-aa17-47d5-9d7e-0672706c1847-secret-metrics-server-client-certs\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.089153 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.089079    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6f417180-aa17-47d5-9d7e-0672706c1847-secret-metrics-server-tls\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.147117 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.147088    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjb6l\" (UniqueName: \"kubernetes.io/projected/6f417180-aa17-47d5-9d7e-0672706c1847-kube-api-access-gjb6l\") pod \"metrics-server-f99b99bd4-fqbcv\" (UID: \"6f417180-aa17-47d5-9d7e-0672706c1847\") " pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.175259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.175197    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cfzh5" podStartSLOduration=3.579627791 podStartE2EDuration="6.175179461s" totalImagesPullingTime="2.51109984s" totalInitContainerRuntime="84.45183ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:29.174565864 +0000 UTC m=+36.385783009" watchObservedRunningTime="2026-10-30 19:16:29.175179461 +0000 UTC m=+36.386396604"
Oct 30 19:16:29.240810 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.240749    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:29.677262 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.677229    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f99b99bd4-fqbcv"]
Oct 30 19:16:29.680498 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:29.680474    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f417180_aa17_47d5_9d7e_0672706c1847.slice/crio-559bf9c89095cf7d9e4bc8fef31291c84e5421442d32e73d73c323517c78c599 WatchSource:0}: Error finding container 559bf9c89095cf7d9e4bc8fef31291c84e5421442d32e73d73c323517c78c599: Status 404 returned error can't find the container with id 559bf9c89095cf7d9e4bc8fef31291c84e5421442d32e73d73c323517c78c599
Oct 30 19:16:29.719830 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.719789    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"]
Oct 30 19:16:29.721955 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.721939    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.782490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.782437    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 19:16:29.782690 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.782671    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 19:16:29.782760 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.782689    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 19:16:29.782760 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.782703    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 19:16:29.782978 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.782962    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mzxsf\""
Oct 30 19:16:29.783048 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.782973    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 19:16:29.791807 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.791789    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-federate-client-tls\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.791938 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.791895    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-secret-telemeter-client\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.791938 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.791918    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rhm7\" (UniqueName: \"kubernetes.io/projected/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-kube-api-access-7rhm7\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.792034 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.791948    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-telemeter-client-tls\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.792078 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.792033    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.792160 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.792141    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-serving-certs-ca-bundle\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.792222 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.792192    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-metrics-client-ca\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.792282 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.792257    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.832895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.832875    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 19:16:29.833730 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.833714    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"]
Oct 30 19:16:29.893164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.893138    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.893282 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.893168    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-serving-certs-ca-bundle\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.893282 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.893192    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-metrics-client-ca\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.893282 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.893208    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.893282 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.893248    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-federate-client-tls\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.893489 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.893295    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-secret-telemeter-client\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.893489 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.893325    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rhm7\" (UniqueName: \"kubernetes.io/projected/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-kube-api-access-7rhm7\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.893489 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.893378    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-telemeter-client-tls\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.894015 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.893956    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-serving-certs-ca-bundle\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.894255 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.894197    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-metrics-client-ca\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.894449 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.894340    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.895842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.895796    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-federate-client-tls\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.895940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.895849    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.895940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.895924    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-secret-telemeter-client\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.896050 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.896014    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-telemeter-client-tls\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:29.950600 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:29.950577    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rhm7\" (UniqueName: \"kubernetes.io/projected/2eb5de2d-6c5f-4918-a6d6-0e7a86287484-kube-api-access-7rhm7\") pod \"telemeter-client-54bbb6dd67-c6z85\" (UID: \"2eb5de2d-6c5f-4918-a6d6-0e7a86287484\") " pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:30.033300 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.033250    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"
Oct 30 19:16:30.059525 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.059495    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv" event={"ID":"6f417180-aa17-47d5-9d7e-0672706c1847","Type":"ContainerStarted","Data":"559bf9c89095cf7d9e4bc8fef31291c84e5421442d32e73d73c323517c78c599"}
Oct 30 19:16:30.060991 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.060965    4420 generic.go:356] "Generic (PLEG): container finished" podID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerID="87e330d3af579f2b65fdde0f5f7f3e2fdee266eb6b7f0714162c4189faecaafc" exitCode=0
Oct 30 19:16:30.061089 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.061059    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerDied","Data":"87e330d3af579f2b65fdde0f5f7f3e2fdee266eb6b7f0714162c4189faecaafc"}
Oct 30 19:16:30.429514 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.429485    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-p59mt"]
Oct 30 19:16:30.432378 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.432361    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-p59mt"
Oct 30 19:16:30.455130 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.455108    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54bbb6dd67-c6z85"]
Oct 30 19:16:30.457597 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:30.457573    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb5de2d_6c5f_4918_a6d6_0e7a86287484.slice/crio-09430acfeac4690409c5ac57a4c7e678622819a7a05855f95f326ce12a76e049 WatchSource:0}: Error finding container 09430acfeac4690409c5ac57a4c7e678622819a7a05855f95f326ce12a76e049: Status 404 returned error can't find the container with id 09430acfeac4690409c5ac57a4c7e678622819a7a05855f95f326ce12a76e049
Oct 30 19:16:30.485318 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.485172    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-2kht4\""
Oct 30 19:16:30.485458 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.485325    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 19:16:30.485840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.485801    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 19:16:30.494461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.494428    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-p59mt"]
Oct 30 19:16:30.497549 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.497524    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df08ad8c-6d14-4c14-96d6-c2f93a144445-tmp\") pod \"downloads-6dd789c8b7-p59mt\" (UID: \"df08ad8c-6d14-4c14-96d6-c2f93a144445\") " pod="openshift-console/downloads-6dd789c8b7-p59mt"
Oct 30 19:16:30.497699 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.497675    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxwb2\" (UniqueName: \"kubernetes.io/projected/df08ad8c-6d14-4c14-96d6-c2f93a144445-kube-api-access-jxwb2\") pod \"downloads-6dd789c8b7-p59mt\" (UID: \"df08ad8c-6d14-4c14-96d6-c2f93a144445\") " pod="openshift-console/downloads-6dd789c8b7-p59mt"
Oct 30 19:16:30.599025 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.598993    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxwb2\" (UniqueName: \"kubernetes.io/projected/df08ad8c-6d14-4c14-96d6-c2f93a144445-kube-api-access-jxwb2\") pod \"downloads-6dd789c8b7-p59mt\" (UID: \"df08ad8c-6d14-4c14-96d6-c2f93a144445\") " pod="openshift-console/downloads-6dd789c8b7-p59mt"
Oct 30 19:16:30.599198 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.599049    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df08ad8c-6d14-4c14-96d6-c2f93a144445-tmp\") pod \"downloads-6dd789c8b7-p59mt\" (UID: \"df08ad8c-6d14-4c14-96d6-c2f93a144445\") " pod="openshift-console/downloads-6dd789c8b7-p59mt"
Oct 30 19:16:30.599404 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.599374    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df08ad8c-6d14-4c14-96d6-c2f93a144445-tmp\") pod \"downloads-6dd789c8b7-p59mt\" (UID: \"df08ad8c-6d14-4c14-96d6-c2f93a144445\") " pod="openshift-console/downloads-6dd789c8b7-p59mt"
Oct 30 19:16:30.656288 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.656259    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxwb2\" (UniqueName: \"kubernetes.io/projected/df08ad8c-6d14-4c14-96d6-c2f93a144445-kube-api-access-jxwb2\") pod \"downloads-6dd789c8b7-p59mt\" (UID: \"df08ad8c-6d14-4c14-96d6-c2f93a144445\") " pod="openshift-console/downloads-6dd789c8b7-p59mt"
Oct 30 19:16:30.742576 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:30.742519    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-p59mt"
Oct 30 19:16:31.065622 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:31.065547    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85" event={"ID":"2eb5de2d-6c5f-4918-a6d6-0e7a86287484","Type":"ContainerStarted","Data":"09430acfeac4690409c5ac57a4c7e678622819a7a05855f95f326ce12a76e049"}
Oct 30 19:16:31.116565 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:31.116377    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-p59mt"]
Oct 30 19:16:31.120230 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:31.120181    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf08ad8c_6d14_4c14_96d6_c2f93a144445.slice/crio-d137fc6ca1ad6fdc6be63c6b5fdaac8951316613016de44ca7009a5d91963732 WatchSource:0}: Error finding container d137fc6ca1ad6fdc6be63c6b5fdaac8951316613016de44ca7009a5d91963732: Status 404 returned error can't find the container with id d137fc6ca1ad6fdc6be63c6b5fdaac8951316613016de44ca7009a5d91963732
Oct 30 19:16:32.076312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:32.076270    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv" event={"ID":"6f417180-aa17-47d5-9d7e-0672706c1847","Type":"ContainerStarted","Data":"1182bdf96adfefda8190b8417bad68a9703dabb1f5ff2af46a9f9bf73707886d"}
Oct 30 19:16:32.078341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:32.078310    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-p59mt" event={"ID":"df08ad8c-6d14-4c14-96d6-c2f93a144445","Type":"ContainerStarted","Data":"d137fc6ca1ad6fdc6be63c6b5fdaac8951316613016de44ca7009a5d91963732"}
Oct 30 19:16:32.195624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:32.194555    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv" podStartSLOduration=2.325328541 podStartE2EDuration="4.194534982s" totalImagesPullingTime="1.869206441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:32.193090939 +0000 UTC m=+39.404308084" watchObservedRunningTime="2026-10-30 19:16:32.194534982 +0000 UTC m=+39.405752128"
Oct 30 19:16:33.085576 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.085541    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerStarted","Data":"b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526"}
Oct 30 19:16:33.085983 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.085615    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerStarted","Data":"352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1"}
Oct 30 19:16:33.085983 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.085648    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerStarted","Data":"2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690"}
Oct 30 19:16:33.085983 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.085676    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerStarted","Data":"138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f"}
Oct 30 19:16:33.085983 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.085705    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerStarted","Data":"ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03"}
Oct 30 19:16:33.088367 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.087843    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85" event={"ID":"2eb5de2d-6c5f-4918-a6d6-0e7a86287484","Type":"ContainerStarted","Data":"f81f2894c82cd41b3fd35bce1523be38fd23a474ff391668ddd8c378046aa696"}
Oct 30 19:16:33.088367 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.087895    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85" event={"ID":"2eb5de2d-6c5f-4918-a6d6-0e7a86287484","Type":"ContainerStarted","Data":"c59451825504bbda5d882a092b3c7df0bab058a2f26197b313af4039d12852f1"}
Oct 30 19:16:33.088367 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.087916    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85" event={"ID":"2eb5de2d-6c5f-4918-a6d6-0e7a86287484","Type":"ContainerStarted","Data":"1ed8b16eed54df2746de7f70537130726d8da3a4ccfa663d9f94895e718d53a7"}
Oct 30 19:16:33.191767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.191694    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:16:33.196847 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.196545    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225572 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225541    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225587    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfjnp\" (UniqueName: \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-kube-api-access-qfjnp\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225612    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225634    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225667    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225689    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225715    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225745    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-config\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225768    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225790    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225827    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225857    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.225905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225881    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.226098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225917    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.226098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225949    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.226098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.225981    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.226098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.226026    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.226098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.226054    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.226098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.226075    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.226098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.226096    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.255665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.255188    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-54bbb6dd67-c6z85" podStartSLOduration=2.048661967 podStartE2EDuration="4.255175027s" totalImagesPullingTime="2.20651306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:33.252846872 +0000 UTC m=+40.464064010" watchObservedRunningTime="2026-10-30 19:16:33.255175027 +0000 UTC m=+40.466392171"
Oct 30 19:16:33.256122 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.255909    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 19:16:33.256122 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.255956    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 19:16:33.256122 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.255966    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 19:16:33.256122 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.256097    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 19:16:33.256339 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.256146    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 19:16:33.297328 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.297304    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 19:16:33.297453 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.297347    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c2olh7le6h51h\""
Oct 30 19:16:33.297453 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.297373    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 19:16:33.297549 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.297465    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 19:16:33.301777 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.301757    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 19:16:33.306411 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.306387    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 19:16:33.327381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.327357    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.327500 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.327404    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.327500 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.327459    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.327500 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.327491    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.328306 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.327872    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.328452 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.328426    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.328618 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.328482    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.328692 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.328675    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.328754 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.328714    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfjnp\" (UniqueName: \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-kube-api-access-qfjnp\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.328877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.328858    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.328941 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.328898    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.328991 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.328941    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.328991 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.328972    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.329089 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.329011    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.329140 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.329099    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-config\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.329140 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.329131    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.329237 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.329217    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.329293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.329247    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.329293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.329285    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.329389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.329322    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.330474 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.330448    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.330768 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.330751    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.332899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.331590    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.332899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.331884    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.332899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.332290    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.332899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.332567    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.332899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.332586    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.332899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.332679    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-config-out\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.332899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.332858    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.336052 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.336034    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.337153 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.336737    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.337153 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.337135    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.337270 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.337135    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.337884 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.337807    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-config\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.338497 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.338480    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-web-config\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.354008 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.353757    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 19:16:33.354008 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.353968    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 19:16:33.354008 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.353966    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 19:16:33.354203 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.354011    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jr2cw\""
Oct 30 19:16:33.354203 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.354151    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 19:16:33.355325 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.355305    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:16:33.360703 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.360672    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.365111 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.365088    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.365230 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.365210    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.365542 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.365524    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.389307 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.389260    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfjnp\" (UniqueName: \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-kube-api-access-qfjnp\") pod \"prometheus-k8s-0\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.519103 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.519025    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:33.978876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:33.978851    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:16:33.981230 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:16:33.981197    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb09a395_3eaa_4630_a12e_53eae7aff574.slice/crio-11919287edfe85c537715c61340352f61b3c3dc1d1408924e220a301af9db982 WatchSource:0}: Error finding container 11919287edfe85c537715c61340352f61b3c3dc1d1408924e220a301af9db982: Status 404 returned error can't find the container with id 11919287edfe85c537715c61340352f61b3c3dc1d1408924e220a301af9db982
Oct 30 19:16:34.097178 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:34.097142    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerStarted","Data":"97bd12dfa98151b209af344e0ee2dc0b2105cbd687656f4f6a58719aa25c6a13"}
Oct 30 19:16:34.097559 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:34.097230    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerStarted","Data":"11919287edfe85c537715c61340352f61b3c3dc1d1408924e220a301af9db982"}
Oct 30 19:16:35.110392 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:35.110308    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerStarted","Data":"203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf"}
Oct 30 19:16:35.113173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:35.113141    4420 generic.go:356] "Generic (PLEG): container finished" podID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerID="97bd12dfa98151b209af344e0ee2dc0b2105cbd687656f4f6a58719aa25c6a13" exitCode=0
Oct 30 19:16:35.113303 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:35.113210    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerDied","Data":"97bd12dfa98151b209af344e0ee2dc0b2105cbd687656f4f6a58719aa25c6a13"}
Oct 30 19:16:35.205468 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:16:35.205429    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:16:35.243794 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:35.243748    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.191311 podStartE2EDuration="9.243733385s" totalImagesPullingTime="6.041139262s" totalInitContainerRuntime="11.283123ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:35.242095414 +0000 UTC m=+42.453312589" watchObservedRunningTime="2026-10-30 19:16:35.243733385 +0000 UTC m=+42.454950529"
Oct 30 19:16:39.134304 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:39.134223    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerStarted","Data":"71badad9dad6655b936f9bd782f83de8aa43ba9872e9b122522dbe74509149b5"}
Oct 30 19:16:39.134668 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:39.134302    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerStarted","Data":"857387c96b1921b52ebab30f936cd9b680ba24241573eb5feef02aea1096b2ed"}
Oct 30 19:16:42.157377 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:42.157341    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerStarted","Data":"3f12848c2706321a84619806c239e977c5d74658d30f8ce474ef2105c0ae110d"}
Oct 30 19:16:42.157789 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:42.157434    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerStarted","Data":"49fdec7d32b49dc61e0a7ef79f3f72ad21db3f87047662a14b07fb8f5ef6239e"}
Oct 30 19:16:42.157789 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:42.157466    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerStarted","Data":"90d49b5946468dd746de58791f21b916e15a9b6b6ac16e698785626c0973b573"}
Oct 30 19:16:42.157789 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:42.157494    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerStarted","Data":"d6e50c4da418ec44c5c3406b95a96990200cac8bbc03371de42257c569a63b84"}
Oct 30 19:16:42.336674 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:42.336566    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.9155479610000001 podStartE2EDuration="9.336549383s" totalImagesPullingTime="7.404427208s" totalInitContainerRuntime="16.574214ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:42.335070541 +0000 UTC m=+49.546287687" watchObservedRunningTime="2026-10-30 19:16:42.336549383 +0000 UTC m=+49.547766527"
Oct 30 19:16:43.519425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:43.519386    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:16:49.189492 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:49.189448    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-p59mt" event={"ID":"df08ad8c-6d14-4c14-96d6-c2f93a144445","Type":"ContainerStarted","Data":"46c2977d9790ad806777d608bc8b79288b2acf2edef5d95795bf7acf495e8d1c"}
Oct 30 19:16:49.190038 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:49.189658    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-p59mt"
Oct 30 19:16:49.203343 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:49.203320    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-p59mt"
Oct 30 19:16:49.241317 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:49.241292    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:49.241972 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:49.241953    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:16:49.404914 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:16:49.404858    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-p59mt" podStartSLOduration=1.758773658 podStartE2EDuration="19.404839494s" totalImagesPullingTime="17.646065836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:16:49.403534479 +0000 UTC m=+56.614751648" watchObservedRunningTime="2026-10-30 19:16:49.404839494 +0000 UTC m=+56.616056641"
Oct 30 19:17:05.194698 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:17:05.194663    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:17:09.246397 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:09.246370    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:17:09.250045 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:09.250024    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f99b99bd4-fqbcv"
Oct 30 19:17:35.195430 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:17:35.195394    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:17:37.941390 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:37.941359    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:37.959633 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:37.959612    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:38.387152 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:38.387132    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:50.579658 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:50.579622    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:17:50.580379 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:50.580348    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="alertmanager" containerID="cri-o://ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03" gracePeriod=120
Oct 30 19:17:50.580479 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:50.580367    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy-metric" containerID="cri-o://b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526" gracePeriod=120
Oct 30 19:17:50.580479 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:50.580437    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy-web" containerID="cri-o://2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690" gracePeriod=120
Oct 30 19:17:50.580591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:50.580472    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="prom-label-proxy" containerID="cri-o://203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf" gracePeriod=120
Oct 30 19:17:50.580591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:50.580446    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy" containerID="cri-o://352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1" gracePeriod=120
Oct 30 19:17:50.580591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:50.580481    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="config-reloader" containerID="cri-o://138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f" gracePeriod=120
Oct 30 19:17:51.420917 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.420886    4420 generic.go:356] "Generic (PLEG): container finished" podID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerID="203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf" exitCode=0
Oct 30 19:17:51.420917 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.420909    4420 generic.go:356] "Generic (PLEG): container finished" podID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerID="b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526" exitCode=0
Oct 30 19:17:51.420917 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.420915    4420 generic.go:356] "Generic (PLEG): container finished" podID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerID="352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1" exitCode=0
Oct 30 19:17:51.420917 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.420921    4420 generic.go:356] "Generic (PLEG): container finished" podID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerID="138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f" exitCode=0
Oct 30 19:17:51.421166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.420927    4420 generic.go:356] "Generic (PLEG): container finished" podID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerID="ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03" exitCode=0
Oct 30 19:17:51.421166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.420955    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerDied","Data":"203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf"}
Oct 30 19:17:51.421166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.421025    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerDied","Data":"b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526"}
Oct 30 19:17:51.421166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.421050    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerDied","Data":"352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1"}
Oct 30 19:17:51.421166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.421073    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerDied","Data":"138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f"}
Oct 30 19:17:51.421166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.421093    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerDied","Data":"ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03"}
Oct 30 19:17:51.827330 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.827308    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:51.935201 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935145    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-main-tls\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.935201 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935174    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-kube-api-access-rs8hv\" (UniqueName: \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-kube-api-access-rs8hv\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.935201 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935192    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-web\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.935437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935210    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.935437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935227    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-trusted-ca-bundle\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.935437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935280    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-out\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.935437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935304    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-web-config\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-web-config\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.935437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935334    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-tls-assets\" (UniqueName: \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-tls-assets\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.935437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935367    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-volume\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-volume\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.935437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935395    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.935437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935435    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-cluster-tls-config\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.936053 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935487    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-metrics-client-ca\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.936053 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.935534    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-main-db\") pod \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\" (UID: \"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca\") "
Oct 30 19:17:51.936279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.936250    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-main-db" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:51.936502 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.936476    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-metrics-client-ca" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:51.936589 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.936526    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-trusted-ca-bundle" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:51.938577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.938532    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-out" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:51.938577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.938543    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-kube-api-access-rs8hv" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "kube-api-access-rs8hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:51.938739 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.938710    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:51.938959 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.938932    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-main-tls" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:51.939066 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.939040    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-volume" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:51.939184 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.939163    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-web" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:51.939227 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.939180    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-tls-assets" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:51.939361 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.939339    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-metric" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:51.943879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.943844    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-cluster-tls-config" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:51.950297 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:51.950279    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-web-config" pod "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" (UID: "a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:52.036689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036668    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-metrics-client-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.036689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036688    4420 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-main-db\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.036799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036705    4420 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-main-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.036799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036715    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs8hv\" (UniqueName: \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-kube-api-access-rs8hv\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.036799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036725    4420 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.036799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036734    4420 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.036799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036742    4420 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.036799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036751    4420 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-out\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.036799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036781    4420 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-web-config\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.036799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036793    4420 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-tls-assets\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.036799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036801    4420 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-config-volume\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.037062 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036809    4420 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.037062 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.036836    4420 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca-cluster-tls-config\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:52.431687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.431650    4420 generic.go:356] "Generic (PLEG): container finished" podID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerID="2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690" exitCode=0
Oct 30 19:17:52.431859 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.431738    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.431859 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.431733    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerDied","Data":"2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690"}
Oct 30 19:17:52.431938 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.431902    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca","Type":"ContainerDied","Data":"3463e58f1dec6efa8e41da415a14272771351b5ed015341880eb697b6e20d481"}
Oct 30 19:17:52.431938 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.431936    4420 scope.go:122] "RemoveContainer" containerID="203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf"
Oct 30 19:17:52.440569 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.440554    4420 scope.go:122] "RemoveContainer" containerID="b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526"
Oct 30 19:17:52.447250 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.447236    4420 scope.go:122] "RemoveContainer" containerID="352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1"
Oct 30 19:17:52.453666 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.453654    4420 scope.go:122] "RemoveContainer" containerID="2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690"
Oct 30 19:17:52.460224 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.460209    4420 scope.go:122] "RemoveContainer" containerID="138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f"
Oct 30 19:17:52.466862 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.466847    4420 scope.go:122] "RemoveContainer" containerID="ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03"
Oct 30 19:17:52.473414 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.473401    4420 scope.go:122] "RemoveContainer" containerID="87e330d3af579f2b65fdde0f5f7f3e2fdee266eb6b7f0714162c4189faecaafc"
Oct 30 19:17:52.479907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.479891    4420 scope.go:122] "RemoveContainer" containerID="203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf"
Oct 30 19:17:52.480159 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:17:52.480139    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf\": container with ID starting with 203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf not found: ID does not exist" containerID="203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf"
Oct 30 19:17:52.480204 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.480167    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf"} err="failed to get container status \"203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf\": rpc error: code = NotFound desc = could not find container \"203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf\": container with ID starting with 203028de3603b1decbc65327fd4543ad9c3a46c7e125a4c582d2a6b8ea446ccf not found: ID does not exist"
Oct 30 19:17:52.480204 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.480197    4420 scope.go:122] "RemoveContainer" containerID="b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526"
Oct 30 19:17:52.480415 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:17:52.480400    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526\": container with ID starting with b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526 not found: ID does not exist" containerID="b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526"
Oct 30 19:17:52.480451 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.480420    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526"} err="failed to get container status \"b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526\": rpc error: code = NotFound desc = could not find container \"b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526\": container with ID starting with b21f1c5c5174478c1e07a1196d93f1b80d2f40900b233930a676f85da0ae8526 not found: ID does not exist"
Oct 30 19:17:52.480451 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.480438    4420 scope.go:122] "RemoveContainer" containerID="352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1"
Oct 30 19:17:52.480638 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:17:52.480622    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1\": container with ID starting with 352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1 not found: ID does not exist" containerID="352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1"
Oct 30 19:17:52.480678 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.480641    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1"} err="failed to get container status \"352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1\": rpc error: code = NotFound desc = could not find container \"352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1\": container with ID starting with 352b2bd49168a160da71fef7c19e0dea36057e8a6e5299a17823879c2ea994f1 not found: ID does not exist"
Oct 30 19:17:52.480678 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.480652    4420 scope.go:122] "RemoveContainer" containerID="2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690"
Oct 30 19:17:52.480883 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:17:52.480866    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690\": container with ID starting with 2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690 not found: ID does not exist" containerID="2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690"
Oct 30 19:17:52.480934 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.480888    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690"} err="failed to get container status \"2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690\": rpc error: code = NotFound desc = could not find container \"2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690\": container with ID starting with 2ae8216bf4025c3ab767fe667fc56c32fa6f2f98303fa19b9b01924dbcc2f690 not found: ID does not exist"
Oct 30 19:17:52.480934 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.480903    4420 scope.go:122] "RemoveContainer" containerID="138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f"
Oct 30 19:17:52.481117 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:17:52.481103    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f\": container with ID starting with 138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f not found: ID does not exist" containerID="138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f"
Oct 30 19:17:52.481159 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.481122    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f"} err="failed to get container status \"138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f\": rpc error: code = NotFound desc = could not find container \"138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f\": container with ID starting with 138555428ce3afbbdda3f3fa13bff909cca1e0899de28aeb61bbd16455e8bb5f not found: ID does not exist"
Oct 30 19:17:52.481159 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.481133    4420 scope.go:122] "RemoveContainer" containerID="ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03"
Oct 30 19:17:52.481319 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:17:52.481305    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03\": container with ID starting with ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03 not found: ID does not exist" containerID="ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03"
Oct 30 19:17:52.481356 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.481322    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03"} err="failed to get container status \"ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03\": rpc error: code = NotFound desc = could not find container \"ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03\": container with ID starting with ec56155342dab1274c3051639a48bf8a7dc84641924713c3bf63dbf1bdd3ec03 not found: ID does not exist"
Oct 30 19:17:52.481356 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.481332    4420 scope.go:122] "RemoveContainer" containerID="87e330d3af579f2b65fdde0f5f7f3e2fdee266eb6b7f0714162c4189faecaafc"
Oct 30 19:17:52.481555 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:17:52.481541    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87e330d3af579f2b65fdde0f5f7f3e2fdee266eb6b7f0714162c4189faecaafc\": container with ID starting with 87e330d3af579f2b65fdde0f5f7f3e2fdee266eb6b7f0714162c4189faecaafc not found: ID does not exist" containerID="87e330d3af579f2b65fdde0f5f7f3e2fdee266eb6b7f0714162c4189faecaafc"
Oct 30 19:17:52.481597 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.481558    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87e330d3af579f2b65fdde0f5f7f3e2fdee266eb6b7f0714162c4189faecaafc"} err="failed to get container status \"87e330d3af579f2b65fdde0f5f7f3e2fdee266eb6b7f0714162c4189faecaafc\": rpc error: code = NotFound desc = could not find container \"87e330d3af579f2b65fdde0f5f7f3e2fdee266eb6b7f0714162c4189faecaafc\": container with ID starting with 87e330d3af579f2b65fdde0f5f7f3e2fdee266eb6b7f0714162c4189faecaafc not found: ID does not exist"
Oct 30 19:17:52.610934 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.610909    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:17:52.660982 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.660959    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:17:52.677485 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677462    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:17:52.677737 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677726    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy-web"
Oct 30 19:17:52.677782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677739    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy-web"
Oct 30 19:17:52.677782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677750    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="prom-label-proxy"
Oct 30 19:17:52.677782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677754    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="prom-label-proxy"
Oct 30 19:17:52.677782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677765    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy"
Oct 30 19:17:52.677782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677770    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy"
Oct 30 19:17:52.677782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677777    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="config-reloader"
Oct 30 19:17:52.677782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677782    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="config-reloader"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677793    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="init-config-reloader"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677798    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="init-config-reloader"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677824    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="alertmanager"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677831    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="alertmanager"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677837    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy-metric"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677841    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy-metric"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677890    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="alertmanager"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677897    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy-metric"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677905    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy-web"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677911    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="prom-label-proxy"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677916    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="kube-rbac-proxy"
Oct 30 19:17:52.677995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.677921    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" containerName="config-reloader"
Oct 30 19:17:52.681449 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.681431    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.740916 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.740895    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3fcfbfee-55e6-4a59-8900-834abb558ae4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741017 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.740929    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-config-volume\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741017 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.740950    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741017 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.740966    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zt6p\" (UniqueName: \"kubernetes.io/projected/3fcfbfee-55e6-4a59-8900-834abb558ae4-kube-api-access-9zt6p\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.741056    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.741110    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fcfbfee-55e6-4a59-8900-834abb558ae4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.741149    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fcfbfee-55e6-4a59-8900-834abb558ae4-config-out\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741311 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.741185    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741311 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.741222    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-web-config\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741311 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.741303    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741456 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.741333    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741456 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.741383    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fcfbfee-55e6-4a59-8900-834abb558ae4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.741456 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.741415    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fcfbfee-55e6-4a59-8900-834abb558ae4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.784231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.784183    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 19:17:52.784231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.784193    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 19:17:52.784231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.784187    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 19:17:52.784231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.784196    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 19:17:52.784231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.784220    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 19:17:52.784231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.784229    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 19:17:52.784231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.784236    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 19:17:52.784231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.784229    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 19:17:52.784530 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.784186    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 19:17:52.784530 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.784286    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d724h\""
Oct 30 19:17:52.791082 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.791059    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:17:52.842309 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842287    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.842681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842319    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fcfbfee-55e6-4a59-8900-834abb558ae4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.842681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842338    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fcfbfee-55e6-4a59-8900-834abb558ae4-config-out\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.842681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842452    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.842681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842501    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-web-config\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.842681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842534    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.842681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842567    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.842681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842603    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fcfbfee-55e6-4a59-8900-834abb558ae4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.842681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842634    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fcfbfee-55e6-4a59-8900-834abb558ae4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.842681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842676    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3fcfbfee-55e6-4a59-8900-834abb558ae4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.843142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842715    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-config-volume\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.843142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842750    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.843142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.842776    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zt6p\" (UniqueName: \"kubernetes.io/projected/3fcfbfee-55e6-4a59-8900-834abb558ae4-kube-api-access-9zt6p\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.844131 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.844104    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fcfbfee-55e6-4a59-8900-834abb558ae4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.844236 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.844218    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3fcfbfee-55e6-4a59-8900-834abb558ae4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.844304 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.844286    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fcfbfee-55e6-4a59-8900-834abb558ae4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.845937 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.845809    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fcfbfee-55e6-4a59-8900-834abb558ae4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.846109 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.846064    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.846220 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.846190    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fcfbfee-55e6-4a59-8900-834abb558ae4-config-out\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.846288 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.846217    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-web-config\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.846459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.846440    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.846852 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.846833    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.847141 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.847121    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.847192 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.847162    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.847567 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.847548    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3fcfbfee-55e6-4a59-8900-834abb558ae4-config-volume\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:52.893641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.893616    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca" path="/var/lib/kubelet/pods/a14d5a8d-d60b-4c39-b6a7-3a3b9f2f05ca/volumes"
Oct 30 19:17:52.900618 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:52.900596    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zt6p\" (UniqueName: \"kubernetes.io/projected/3fcfbfee-55e6-4a59-8900-834abb558ae4-kube-api-access-9zt6p\") pod \"alertmanager-main-0\" (UID: \"3fcfbfee-55e6-4a59-8900-834abb558ae4\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:53.053026 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:53.052973    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-d724h\""
Oct 30 19:17:53.063044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:53.063021    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 19:17:53.535702 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:53.535668    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 19:17:53.538304 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:17:53.538276    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fcfbfee_55e6_4a59_8900_834abb558ae4.slice/crio-61b80ded9f2241306fa2fce31e0fc82cc69c9b4acd0e5272853e9f2a11a02312 WatchSource:0}: Error finding container 61b80ded9f2241306fa2fce31e0fc82cc69c9b4acd0e5272853e9f2a11a02312: Status 404 returned error can't find the container with id 61b80ded9f2241306fa2fce31e0fc82cc69c9b4acd0e5272853e9f2a11a02312
Oct 30 19:17:54.441062 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:54.441027    4420 generic.go:356] "Generic (PLEG): container finished" podID="3fcfbfee-55e6-4a59-8900-834abb558ae4" containerID="71046656ffde535d2511f185be4f5ff77097948145de206a737fac428cdf0d4c" exitCode=0
Oct 30 19:17:54.441492 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:54.441090    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fcfbfee-55e6-4a59-8900-834abb558ae4","Type":"ContainerDied","Data":"71046656ffde535d2511f185be4f5ff77097948145de206a737fac428cdf0d4c"}
Oct 30 19:17:54.441492 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:54.441149    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fcfbfee-55e6-4a59-8900-834abb558ae4","Type":"ContainerStarted","Data":"61b80ded9f2241306fa2fce31e0fc82cc69c9b4acd0e5272853e9f2a11a02312"}
Oct 30 19:17:55.448670 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:55.448634    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fcfbfee-55e6-4a59-8900-834abb558ae4","Type":"ContainerStarted","Data":"d9025928f286750242cda684633708288cf32be331198ca52c6d3f81679f64d8"}
Oct 30 19:17:55.449054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:55.448713    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fcfbfee-55e6-4a59-8900-834abb558ae4","Type":"ContainerStarted","Data":"8797682c51b221926b538d737167436a481b375fd8fded8d27a03a2eec64d45b"}
Oct 30 19:17:55.449054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:55.448754    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fcfbfee-55e6-4a59-8900-834abb558ae4","Type":"ContainerStarted","Data":"803d6d7abf1e75dea59c41de13363a062159c84fe271e62500f5237b99bc211e"}
Oct 30 19:17:55.449054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:55.448789    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fcfbfee-55e6-4a59-8900-834abb558ae4","Type":"ContainerStarted","Data":"1651353777bccfe524355c3624ad2a049dbf1f829cd18203137d56845426b32e"}
Oct 30 19:17:55.449054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:55.448848    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fcfbfee-55e6-4a59-8900-834abb558ae4","Type":"ContainerStarted","Data":"a5290a951a43e18eac0441d3877d70b80b26a51554712edfb468dad2837af600"}
Oct 30 19:17:55.449054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:55.448884    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3fcfbfee-55e6-4a59-8900-834abb558ae4","Type":"ContainerStarted","Data":"c070b543e4ce4861677589ea400fc57bda5fb331335a3eafa25730de79d67d4e"}
Oct 30 19:17:55.570430 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:55.570377    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.558101207 podStartE2EDuration="3.570350043s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.248836ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:17:55.569679123 +0000 UTC m=+122.780896267" watchObservedRunningTime="2026-10-30 19:17:55.570350043 +0000 UTC m=+122.781567189"
Oct 30 19:17:56.291308 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.291277    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:17:56.291719 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.291566    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy" containerID="cri-o://49fdec7d32b49dc61e0a7ef79f3f72ad21db3f87047662a14b07fb8f5ef6239e" gracePeriod=600
Oct 30 19:17:56.291719 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.291594    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="thanos-sidecar" containerID="cri-o://d6e50c4da418ec44c5c3406b95a96990200cac8bbc03371de42257c569a63b84" gracePeriod=600
Oct 30 19:17:56.291719 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.291618    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3f12848c2706321a84619806c239e977c5d74658d30f8ce474ef2105c0ae110d" gracePeriod=600
Oct 30 19:17:56.291719 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.291622    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="config-reloader" containerID="cri-o://71badad9dad6655b936f9bd782f83de8aa43ba9872e9b122522dbe74509149b5" gracePeriod=600
Oct 30 19:17:56.291719 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.291546    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="prometheus" containerID="cri-o://857387c96b1921b52ebab30f936cd9b680ba24241573eb5feef02aea1096b2ed" gracePeriod=600
Oct 30 19:17:56.291719 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.291599    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy-web" containerID="cri-o://90d49b5946468dd746de58791f21b916e15a9b6b6ac16e698785626c0973b573" gracePeriod=600
Oct 30 19:17:56.456762 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456725    4420 generic.go:356] "Generic (PLEG): container finished" podID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerID="3f12848c2706321a84619806c239e977c5d74658d30f8ce474ef2105c0ae110d" exitCode=0
Oct 30 19:17:56.456762 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456760    4420 generic.go:356] "Generic (PLEG): container finished" podID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerID="49fdec7d32b49dc61e0a7ef79f3f72ad21db3f87047662a14b07fb8f5ef6239e" exitCode=0
Oct 30 19:17:56.457161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456772    4420 generic.go:356] "Generic (PLEG): container finished" podID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerID="90d49b5946468dd746de58791f21b916e15a9b6b6ac16e698785626c0973b573" exitCode=0
Oct 30 19:17:56.457161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456784    4420 generic.go:356] "Generic (PLEG): container finished" podID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerID="d6e50c4da418ec44c5c3406b95a96990200cac8bbc03371de42257c569a63b84" exitCode=0
Oct 30 19:17:56.457161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456792    4420 generic.go:356] "Generic (PLEG): container finished" podID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerID="71badad9dad6655b936f9bd782f83de8aa43ba9872e9b122522dbe74509149b5" exitCode=0
Oct 30 19:17:56.457161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456806    4420 generic.go:356] "Generic (PLEG): container finished" podID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerID="857387c96b1921b52ebab30f936cd9b680ba24241573eb5feef02aea1096b2ed" exitCode=0
Oct 30 19:17:56.457161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456786    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerDied","Data":"3f12848c2706321a84619806c239e977c5d74658d30f8ce474ef2105c0ae110d"}
Oct 30 19:17:56.457161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456885    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerDied","Data":"49fdec7d32b49dc61e0a7ef79f3f72ad21db3f87047662a14b07fb8f5ef6239e"}
Oct 30 19:17:56.457161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456911    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerDied","Data":"90d49b5946468dd746de58791f21b916e15a9b6b6ac16e698785626c0973b573"}
Oct 30 19:17:56.457161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456935    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerDied","Data":"d6e50c4da418ec44c5c3406b95a96990200cac8bbc03371de42257c569a63b84"}
Oct 30 19:17:56.457161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456957    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerDied","Data":"71badad9dad6655b936f9bd782f83de8aa43ba9872e9b122522dbe74509149b5"}
Oct 30 19:17:56.457161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.456977    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerDied","Data":"857387c96b1921b52ebab30f936cd9b680ba24241573eb5feef02aea1096b2ed"}
Oct 30 19:17:56.547856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.547781    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:56.671889 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.671860    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672016 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.671899    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-kube-rbac-proxy\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672016 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.671925    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-grpc-tls\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672016 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.671949    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-config-out\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672016 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.671978    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-web-config\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-web-config\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672016 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672003    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-trusted-ca-bundle\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672276 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672031    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-1\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672276 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672058    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672360    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-1" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:56.672438 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672389    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-trusted-ca-bundle" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:56.672438 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672411    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-2\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672532 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672466    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-db\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672532 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672507    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-metrics-client-ca\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672625 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672544    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-config\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-config\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.672625 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672577    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-tls-assets\" (UniqueName: \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-tls-assets\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672655    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-kube-api-access-qfjnp\" (UniqueName: \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-kube-api-access-qfjnp\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672681    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-thanos-prometheus-http-client-file\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672720    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-serving-certs-ca-bundle\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672747    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-kubelet-serving-ca-bundle\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672774    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-2" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672783    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-0\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672841    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-tls\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.672868    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-metrics-client-certs\") pod \"bb09a395-3eaa-4630-a12e-53eae7aff574\" (UID: \"bb09a395-3eaa-4630-a12e-53eae7aff574\") "
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.673139    4420 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.673160    4420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.673172    4420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.673894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.673475    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-metrics-client-ca" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:56.674623 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.674530    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-kubelet-serving-ca-bundle" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:56.675397 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.675210    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-serving-certs-ca-bundle" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:56.675397 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.675280    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-grpc-tls" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:56.675397 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.675324    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-kube-rbac-proxy-web" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:56.675397 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.675366    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-kube-rbac-proxy" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:56.675805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.675677    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-0" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:56.675904 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.675877    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-config-out" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:56.676001 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.675962    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-db" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:56.676342 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.676320    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-tls-assets" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:56.676880 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.676810    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-thanos-sidecar-tls" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:56.676880 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.676850    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-tls" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:56.677159 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.677127    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-kube-api-access-qfjnp" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "kube-api-access-qfjnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:56.677284 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.677261    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-config" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:56.677620 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.677599    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-thanos-prometheus-http-client-file" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:56.679609 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.678189    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-metrics-client-certs" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:56.688616 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.688590    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-web-config" pod "bb09a395-3eaa-4630-a12e-53eae7aff574" (UID: "bb09a395-3eaa-4630-a12e-53eae7aff574"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:56.773538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773509    4420 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773531    4420 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-kube-rbac-proxy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773541    4420 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-grpc-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773549    4420 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-config-out\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773558    4420 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-web-config\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773568    4420 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773576    4420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-db\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773583    4420 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-metrics-client-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773592    4420 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-config\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773600    4420 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-tls-assets\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773608    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfjnp\" (UniqueName: \"kubernetes.io/projected/bb09a395-3eaa-4630-a12e-53eae7aff574-kube-api-access-qfjnp\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773616    4420 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773624    4420 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773631    4420 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773639    4420 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bb09a395-3eaa-4630-a12e-53eae7aff574-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773648    4420 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:56.773757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:56.773656    4420 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/bb09a395-3eaa-4630-a12e-53eae7aff574-secret-metrics-client-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:17:57.464046 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.464013    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"bb09a395-3eaa-4630-a12e-53eae7aff574","Type":"ContainerDied","Data":"11919287edfe85c537715c61340352f61b3c3dc1d1408924e220a301af9db982"}
Oct 30 19:17:57.464046 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.464046    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.464549 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.464131    4420 scope.go:122] "RemoveContainer" containerID="3f12848c2706321a84619806c239e977c5d74658d30f8ce474ef2105c0ae110d"
Oct 30 19:17:57.472027 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.472007    4420 scope.go:122] "RemoveContainer" containerID="49fdec7d32b49dc61e0a7ef79f3f72ad21db3f87047662a14b07fb8f5ef6239e"
Oct 30 19:17:57.478749 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.478735    4420 scope.go:122] "RemoveContainer" containerID="90d49b5946468dd746de58791f21b916e15a9b6b6ac16e698785626c0973b573"
Oct 30 19:17:57.485438 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.485419    4420 scope.go:122] "RemoveContainer" containerID="d6e50c4da418ec44c5c3406b95a96990200cac8bbc03371de42257c569a63b84"
Oct 30 19:17:57.491558 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.491539    4420 scope.go:122] "RemoveContainer" containerID="71badad9dad6655b936f9bd782f83de8aa43ba9872e9b122522dbe74509149b5"
Oct 30 19:17:57.497578 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.497551    4420 scope.go:122] "RemoveContainer" containerID="857387c96b1921b52ebab30f936cd9b680ba24241573eb5feef02aea1096b2ed"
Oct 30 19:17:57.504445 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.504429    4420 scope.go:122] "RemoveContainer" containerID="97bd12dfa98151b209af344e0ee2dc0b2105cbd687656f4f6a58719aa25c6a13"
Oct 30 19:17:57.630173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.630106    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:17:57.679821 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.679797    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:17:57.680780 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.680758    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:17:57.681214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681197    4420 cpu_manager.go:414] "containerMap: removing container" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="thanos-sidecar"
Oct 30 19:17:57.681252 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681220    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="thanos-sidecar"
Oct 30 19:17:57.681283 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681251    4420 cpu_manager.go:414] "containerMap: removing container" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy"
Oct 30 19:17:57.681283 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681260    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy"
Oct 30 19:17:57.681283 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681278    4420 cpu_manager.go:414] "containerMap: removing container" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="config-reloader"
Oct 30 19:17:57.681387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681286    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="config-reloader"
Oct 30 19:17:57.681387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681300    4420 cpu_manager.go:414] "containerMap: removing container" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="prometheus"
Oct 30 19:17:57.681387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681307    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="prometheus"
Oct 30 19:17:57.681387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681316    4420 cpu_manager.go:414] "containerMap: removing container" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy-thanos"
Oct 30 19:17:57.681387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681322    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy-thanos"
Oct 30 19:17:57.681387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681340    4420 cpu_manager.go:414] "containerMap: removing container" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="init-config-reloader"
Oct 30 19:17:57.681387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681347    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="init-config-reloader"
Oct 30 19:17:57.681387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681359    4420 cpu_manager.go:414] "containerMap: removing container" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy-web"
Oct 30 19:17:57.681387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681365    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy-web"
Oct 30 19:17:57.681636 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681442    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="config-reloader"
Oct 30 19:17:57.681636 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681453    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="prometheus"
Oct 30 19:17:57.681636 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681461    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy-thanos"
Oct 30 19:17:57.681636 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681471    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy-web"
Oct 30 19:17:57.681636 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681481    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="thanos-sidecar"
Oct 30 19:17:57.681636 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.681491    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" containerName="kube-rbac-proxy"
Oct 30 19:17:57.684891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.684876    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.737833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.737699    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c2olh7le6h51h\""
Oct 30 19:17:57.737833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.737718    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 19:17:57.737833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.737757    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 19:17:57.737833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.737722    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 19:17:57.737833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.737766    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 19:17:57.737833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.737776    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 19:17:57.737833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.737772    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 19:17:57.737833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.737797    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 19:17:57.738199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.737957    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 19:17:57.738199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.737980    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 19:17:57.738199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.738019    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 19:17:57.738199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.738101    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 19:17:57.738199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.738102    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 19:17:57.738199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.738143    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jr2cw\""
Oct 30 19:17:57.782103 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782079    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782111    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782169    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782281 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782208    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782281 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782238    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-config\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782334 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782325    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782363 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782348    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782393 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782366    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782393 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782382    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782453 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782402    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782463    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782518 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782483    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782518 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782504    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kprz\" (UniqueName: \"kubernetes.io/projected/ff787b18-9b9e-412a-bc01-58fca3895706-kube-api-access-4kprz\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782574 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782523    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782602 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782571    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782633 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782600    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782661 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782636    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff787b18-9b9e-412a-bc01-58fca3895706-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782693 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782668    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff787b18-9b9e-412a-bc01-58fca3895706-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782724 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782703    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.782756 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.782722    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.788376 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.788354    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 19:17:57.788505 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.788484    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 19:17:57.883279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883232    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883260    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-config\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883285    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883301    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883319    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883343    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883365    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883395    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883419    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883450    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kprz\" (UniqueName: \"kubernetes.io/projected/ff787b18-9b9e-412a-bc01-58fca3895706-kube-api-access-4kprz\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883482    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883511    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883536    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883556    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff787b18-9b9e-412a-bc01-58fca3895706-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883577    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff787b18-9b9e-412a-bc01-58fca3895706-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883600    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883626    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883660    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883700    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883729    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.883866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883845    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.884363 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.883975    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.884363 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.884115    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.884511 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.884487    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.884801 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.884756    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.886698 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.886674    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-config\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.886790 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.886698    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.887214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.887182    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff787b18-9b9e-412a-bc01-58fca3895706-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.887343 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.887326    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.887554 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.887529    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.888578 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.887662    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.888578 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.887855    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.888578 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.888040    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff787b18-9b9e-412a-bc01-58fca3895706-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.888578 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.888397    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.888776 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.888585    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.889363 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.889341    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff787b18-9b9e-412a-bc01-58fca3895706-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.889513 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.889492    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.889827 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.889799    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.889905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.889888    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff787b18-9b9e-412a-bc01-58fca3895706-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:57.921797 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.921764    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:17:57.943887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:57.943856    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kprz\" (UniqueName: \"kubernetes.io/projected/ff787b18-9b9e-412a-bc01-58fca3895706-kube-api-access-4kprz\") pod \"prometheus-k8s-0\" (UID: \"ff787b18-9b9e-412a-bc01-58fca3895706\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:58.000911 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:58.000873    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:17:58.425642 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:58.425620    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 19:17:58.427342 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:17:58.427309    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff787b18_9b9e_412a_bc01_58fca3895706.slice/crio-e373fb596d0e4da0c9bfb05a56b3b22ebc8ef9317b7017b1181ed13c7b2a5497 WatchSource:0}: Error finding container e373fb596d0e4da0c9bfb05a56b3b22ebc8ef9317b7017b1181ed13c7b2a5497: Status 404 returned error can't find the container with id e373fb596d0e4da0c9bfb05a56b3b22ebc8ef9317b7017b1181ed13c7b2a5497
Oct 30 19:17:58.472527 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:58.472496    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff787b18-9b9e-412a-bc01-58fca3895706","Type":"ContainerStarted","Data":"e373fb596d0e4da0c9bfb05a56b3b22ebc8ef9317b7017b1181ed13c7b2a5497"}
Oct 30 19:17:58.892810 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:58.892778    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb09a395-3eaa-4630-a12e-53eae7aff574" path="/var/lib/kubelet/pods/bb09a395-3eaa-4630-a12e-53eae7aff574/volumes"
Oct 30 19:17:59.478894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:59.478864    4420 generic.go:356] "Generic (PLEG): container finished" podID="ff787b18-9b9e-412a-bc01-58fca3895706" containerID="3128b94ae073a52603c92efcfee08c02ba9183ff389b1d99251e062bbaa0a2a8" exitCode=0
Oct 30 19:17:59.479218 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:17:59.478945    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff787b18-9b9e-412a-bc01-58fca3895706","Type":"ContainerDied","Data":"3128b94ae073a52603c92efcfee08c02ba9183ff389b1d99251e062bbaa0a2a8"}
Oct 30 19:18:00.485774 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:18:00.485738    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff787b18-9b9e-412a-bc01-58fca3895706","Type":"ContainerStarted","Data":"f7e11d35f0d82f02ef2f11a93d84fb6ed23973f12ebf34b624c905c81a52d120"}
Oct 30 19:18:00.486176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:18:00.485802    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff787b18-9b9e-412a-bc01-58fca3895706","Type":"ContainerStarted","Data":"4b8511909fd6c87db38fcfe70456985d0942860e3aa4d4f69db0d4149a80b590"}
Oct 30 19:18:00.486176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:18:00.485851    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff787b18-9b9e-412a-bc01-58fca3895706","Type":"ContainerStarted","Data":"98bfb8aeda2ef1b26a3a7db0014368dc01420816ea4ce1d5df513b0ed109afe7"}
Oct 30 19:18:00.486176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:18:00.485884    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff787b18-9b9e-412a-bc01-58fca3895706","Type":"ContainerStarted","Data":"644e5b1fd943fb9fbe034da0d3a40d042ddc12302a177a5c78857c6ee1756d0e"}
Oct 30 19:18:00.486176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:18:00.485909    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff787b18-9b9e-412a-bc01-58fca3895706","Type":"ContainerStarted","Data":"a1486c6cadd7853c80cee7a41de913110fbb58907ec2dabee8f691bddb631d6a"}
Oct 30 19:18:00.486176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:18:00.485940    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff787b18-9b9e-412a-bc01-58fca3895706","Type":"ContainerStarted","Data":"a9d541ca8e325d5b3cb7a997a664d494e08706091c40be6965676dbdbe654979"}
Oct 30 19:18:00.634435 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:18:00.634389    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.618070541 podStartE2EDuration="3.634373975s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.303434ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:00.634084505 +0000 UTC m=+127.845301660" watchObservedRunningTime="2026-10-30 19:18:00.634373975 +0000 UTC m=+127.845591119"
Oct 30 19:18:03.001703 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:18:03.001675    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:18:05.195316 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:18:05.195245    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:18:35.194973 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:18:35.194933    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:18:58.001426 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:18:58.001397    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:18:58.016960 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:18:58.016939    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:18:58.708307 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:18:58.708281    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 19:19:05.194608 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:19:05.194573    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:19:35.195399 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:19:35.195315    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:20:05.194490 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:20:05.194450    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:20:35.194885 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:20:35.194851    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:20:49.890777 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:49.890744    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"]
Oct 30 19:20:49.893423 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:49.893407    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:49.930644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:49.930617    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d23d6591-7ebf-431f-9dfd-408a973d4a03-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:49.930751 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:49.930646    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7vs\" (UniqueName: \"kubernetes.io/projected/d23d6591-7ebf-431f-9dfd-408a973d4a03-kube-api-access-rr7vs\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:49.930751 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:49.930666    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d23d6591-7ebf-431f-9dfd-408a973d4a03-webhook-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:49.930837 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:49.930773    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d23d6591-7ebf-431f-9dfd-408a973d4a03-metrics-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:49.930874 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:49.930832    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d23d6591-7ebf-431f-9dfd-408a973d4a03-tmp\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:49.930949 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:49.930929    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d23d6591-7ebf-431f-9dfd-408a973d4a03-manifests\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.012972 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.012945    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 19:20:50.013094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.013063    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 19:20:50.013202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.013186    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 19:20:50.013253 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.013196    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 19:20:50.013253 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.013219    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 19:20:50.013253 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.013232    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2h29g\""
Oct 30 19:20:50.026380 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.026358    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"]
Oct 30 19:20:50.031710 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.031690    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d23d6591-7ebf-431f-9dfd-408a973d4a03-metrics-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.031833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.031736    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d23d6591-7ebf-431f-9dfd-408a973d4a03-tmp\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.031833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.031789    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d23d6591-7ebf-431f-9dfd-408a973d4a03-manifests\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.031954 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.031852    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d23d6591-7ebf-431f-9dfd-408a973d4a03-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.031954 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.031879    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7vs\" (UniqueName: \"kubernetes.io/projected/d23d6591-7ebf-431f-9dfd-408a973d4a03-kube-api-access-rr7vs\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.031954 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.031910    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d23d6591-7ebf-431f-9dfd-408a973d4a03-webhook-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.032238 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.032213    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d23d6591-7ebf-431f-9dfd-408a973d4a03-manifests\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.034393 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.034374    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d23d6591-7ebf-431f-9dfd-408a973d4a03-webhook-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.034481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.034417    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d23d6591-7ebf-431f-9dfd-408a973d4a03-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.034575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.034555    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d23d6591-7ebf-431f-9dfd-408a973d4a03-tmp\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.034846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.034808    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d23d6591-7ebf-431f-9dfd-408a973d4a03-metrics-cert\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.107055 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.107034    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7vs\" (UniqueName: \"kubernetes.io/projected/d23d6591-7ebf-431f-9dfd-408a973d4a03-kube-api-access-rr7vs\") pod \"opendatahub-operator-controller-manager-69bfcc8945-xr2np\" (UID: \"d23d6591-7ebf-431f-9dfd-408a973d4a03\") " pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.204986 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.204932    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:50.696554 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:20:50.696521    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23d6591_7ebf_431f_9dfd_408a973d4a03.slice/crio-3b2986ea63cb9898a1065a625fc608b2fef8b767831e1c9b7f20a540149d2e66 WatchSource:0}: Error finding container 3b2986ea63cb9898a1065a625fc608b2fef8b767831e1c9b7f20a540149d2e66: Status 404 returned error can't find the container with id 3b2986ea63cb9898a1065a625fc608b2fef8b767831e1c9b7f20a540149d2e66
Oct 30 19:20:50.742546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:50.742523    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"]
Oct 30 19:20:51.074620 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:51.074558    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np" event={"ID":"d23d6591-7ebf-431f-9dfd-408a973d4a03","Type":"ContainerStarted","Data":"3b2986ea63cb9898a1065a625fc608b2fef8b767831e1c9b7f20a540149d2e66"}
Oct 30 19:20:55.091989 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:55.091953    4420 generic.go:356] "Generic (PLEG): container finished" podID="d23d6591-7ebf-431f-9dfd-408a973d4a03" containerID="849cdc1f7a3006f574252d49be2ad0e8e57424898f081f74ffaa00a1e0ad9896" exitCode=0
Oct 30 19:20:55.092384 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:55.092013    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np" event={"ID":"d23d6591-7ebf-431f-9dfd-408a973d4a03","Type":"ContainerDied","Data":"849cdc1f7a3006f574252d49be2ad0e8e57424898f081f74ffaa00a1e0ad9896"}
Oct 30 19:20:56.097779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:56.097747    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np" event={"ID":"d23d6591-7ebf-431f-9dfd-408a973d4a03","Type":"ContainerStarted","Data":"a3fce76d2580ff4e5ab256a8c73dd467ab9ba5fa5c09144421860d6bcc7fd417"}
Oct 30 19:20:56.098165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:56.097869    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:20:56.324959 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:20:56.324877    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np" podStartSLOduration=2.685691012 podStartE2EDuration="7.324864933s" totalImagesPullingTime="4.197179882s" totalInitContainerRuntime="441.994039ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:20:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:20:56.275617966 +0000 UTC m=+303.486835174" watchObservedRunningTime="2026-10-30 19:20:56.324864933 +0000 UTC m=+303.536082078"
Oct 30 19:21:05.194795 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:21:05.194723    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:21:07.103589 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:07.103558    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-69bfcc8945-xr2np"
Oct 30 19:21:10.662038 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.662007    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"]
Oct 30 19:21:10.666614 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.666595    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.758001 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.757975    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"]
Oct 30 19:21:10.781716 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.781686    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-manager-config\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.781861 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.781720    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tczvm\" (UniqueName: \"kubernetes.io/projected/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-kube-api-access-tczvm\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.781861 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.781741    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-cert\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.781861 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.781830    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-metrics-cert\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.828600 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.828578    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 19:21:10.828600 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.828596    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4gpz6\""
Oct 30 19:21:10.828747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.828604    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 19:21:10.828747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.828611    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 19:21:10.828747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.828579    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 19:21:10.828747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.828577    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 19:21:10.882382 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.882361    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczvm\" (UniqueName: \"kubernetes.io/projected/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-kube-api-access-tczvm\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.882473 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.882392    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-cert\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.882473 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.882424    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-metrics-cert\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.882581 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.882499    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-manager-config\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.883125 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.883103    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-manager-config\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.884852 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.884831    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-metrics-cert\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.884943 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.884859    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-cert\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.950297 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.950243    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczvm\" (UniqueName: \"kubernetes.io/projected/4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41-kube-api-access-tczvm\") pod \"lws-controller-manager-5676f67757-s6dx2\" (UID: \"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41\") " pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:10.977237 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:10.977185    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:11.342871 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:11.342805    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"]
Oct 30 19:21:11.345421 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:21:11.345395    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4d0b3c_b3bc_4e52_9f2e_14e43317ba41.slice/crio-3810d716285f59ad9e8e05a2636876a576cbb4227bdb82785a380d1412fbe5d3 WatchSource:0}: Error finding container 3810d716285f59ad9e8e05a2636876a576cbb4227bdb82785a380d1412fbe5d3: Status 404 returned error can't find the container with id 3810d716285f59ad9e8e05a2636876a576cbb4227bdb82785a380d1412fbe5d3
Oct 30 19:21:11.347231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:11.347217    4420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:21:12.155103 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:12.155067    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2" event={"ID":"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41","Type":"ContainerStarted","Data":"3810d716285f59ad9e8e05a2636876a576cbb4227bdb82785a380d1412fbe5d3"}
Oct 30 19:21:16.172234 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:16.172200    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2" event={"ID":"4c4d0b3c-b3bc-4e52-9f2e-14e43317ba41","Type":"ContainerStarted","Data":"b24c366fc97c39f7c481a232ba75683946176d1e00403c68b44a4e751c8d3200"}
Oct 30 19:21:16.172614 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:16.172328    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:23.313475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.313425    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2" podStartSLOduration=8.889027178 podStartE2EDuration="13.31341069s" totalImagesPullingTime="4.424383512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:16.284991789 +0000 UTC m=+323.496208933" watchObservedRunningTime="2026-10-30 19:21:23.31341069 +0000 UTC m=+330.524627828"
Oct 30 19:21:23.314256 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.314239    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"]
Oct 30 19:21:23.317333 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.317313    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"
Oct 30 19:21:23.381849 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.381801    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a3667f5-8616-4e54-ba3e-f9a530059639-tls-certs\") pod \"kube-auth-proxy-7f6998b7b5-qbhn2\" (UID: \"7a3667f5-8616-4e54-ba3e-f9a530059639\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"
Oct 30 19:21:23.381980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.381864    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjwrz\" (UniqueName: \"kubernetes.io/projected/7a3667f5-8616-4e54-ba3e-f9a530059639-kube-api-access-bjwrz\") pod \"kube-auth-proxy-7f6998b7b5-qbhn2\" (UID: \"7a3667f5-8616-4e54-ba3e-f9a530059639\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"
Oct 30 19:21:23.381980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.381963    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a3667f5-8616-4e54-ba3e-f9a530059639-tmp\") pod \"kube-auth-proxy-7f6998b7b5-qbhn2\" (UID: \"7a3667f5-8616-4e54-ba3e-f9a530059639\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"
Oct 30 19:21:23.418005 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.417967    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zq789\""
Oct 30 19:21:23.418133 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.418076    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 19:21:23.418185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.418129    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 19:21:23.420092 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.420066    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"]
Oct 30 19:21:23.467709 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.467665    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 19:21:23.467863 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.467729    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 19:21:23.482998 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.482974    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a3667f5-8616-4e54-ba3e-f9a530059639-tls-certs\") pod \"kube-auth-proxy-7f6998b7b5-qbhn2\" (UID: \"7a3667f5-8616-4e54-ba3e-f9a530059639\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"
Oct 30 19:21:23.483087 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.483012    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjwrz\" (UniqueName: \"kubernetes.io/projected/7a3667f5-8616-4e54-ba3e-f9a530059639-kube-api-access-bjwrz\") pod \"kube-auth-proxy-7f6998b7b5-qbhn2\" (UID: \"7a3667f5-8616-4e54-ba3e-f9a530059639\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"
Oct 30 19:21:23.483087 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.483073    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a3667f5-8616-4e54-ba3e-f9a530059639-tmp\") pod \"kube-auth-proxy-7f6998b7b5-qbhn2\" (UID: \"7a3667f5-8616-4e54-ba3e-f9a530059639\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"
Oct 30 19:21:23.485307 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.485283    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a3667f5-8616-4e54-ba3e-f9a530059639-tmp\") pod \"kube-auth-proxy-7f6998b7b5-qbhn2\" (UID: \"7a3667f5-8616-4e54-ba3e-f9a530059639\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"
Oct 30 19:21:23.485500 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.485484    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a3667f5-8616-4e54-ba3e-f9a530059639-tls-certs\") pod \"kube-auth-proxy-7f6998b7b5-qbhn2\" (UID: \"7a3667f5-8616-4e54-ba3e-f9a530059639\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"
Oct 30 19:21:23.539335 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.539313    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjwrz\" (UniqueName: \"kubernetes.io/projected/7a3667f5-8616-4e54-ba3e-f9a530059639-kube-api-access-bjwrz\") pod \"kube-auth-proxy-7f6998b7b5-qbhn2\" (UID: \"7a3667f5-8616-4e54-ba3e-f9a530059639\") " pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"
Oct 30 19:21:23.627696 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.627637    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"
Oct 30 19:21:23.995645 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:23.995623    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2"]
Oct 30 19:21:23.998102 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:21:23.998080    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3667f5_8616_4e54_ba3e_f9a530059639.slice/crio-d21aeef7208bddec7a9105c120f095162cffdb518de34357d7dcd8684f19b41f WatchSource:0}: Error finding container d21aeef7208bddec7a9105c120f095162cffdb518de34357d7dcd8684f19b41f: Status 404 returned error can't find the container with id d21aeef7208bddec7a9105c120f095162cffdb518de34357d7dcd8684f19b41f
Oct 30 19:21:24.205274 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:24.205239    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2" event={"ID":"7a3667f5-8616-4e54-ba3e-f9a530059639","Type":"ContainerStarted","Data":"d21aeef7208bddec7a9105c120f095162cffdb518de34357d7dcd8684f19b41f"}
Oct 30 19:21:27.178285 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:27.178262    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5676f67757-s6dx2"
Oct 30 19:21:27.221810 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:27.221777    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2" event={"ID":"7a3667f5-8616-4e54-ba3e-f9a530059639","Type":"ContainerStarted","Data":"7ec76013c018638fe77d613c3c7fad9b694f44d5e273d0353664f586a5cc2c70"}
Oct 30 19:21:27.404370 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:21:27.404320    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f6998b7b5-qbhn2" podStartSLOduration=1.530634719 podStartE2EDuration="4.404302905s" totalImagesPullingTime="2.873668186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:27.403670261 +0000 UTC m=+334.614887408" watchObservedRunningTime="2026-10-30 19:21:27.404302905 +0000 UTC m=+334.615520049"
Oct 30 19:21:35.194629 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:21:35.194593    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:22:05.195832 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:22:05.195786    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:22:35.194272 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:22:35.194193    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:23:05.194166 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:23:05.194133    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:23:35.197109 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:23:35.197068    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:24:05.194739 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:24:05.194645    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:24:33.296907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.296876    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g"]
Oct 30 19:24:33.300720 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.300705    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g"
Oct 30 19:24:33.322008 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.321979    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwfzc\" (UniqueName: \"kubernetes.io/projected/95ba72ee-301b-4265-b57a-013c9f16d8b5-kube-api-access-lwfzc\") pod \"dns-operator-controller-manager-65bfd79598-9c76g\" (UID: \"95ba72ee-301b-4265-b57a-013c9f16d8b5\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g"
Oct 30 19:24:33.360664 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.360634    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 19:24:33.410023 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.409986    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 19:24:33.410023 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.409987    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 19:24:33.410023 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.409996    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-26d48\""
Oct 30 19:24:33.411404 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.411386    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g"]
Oct 30 19:24:33.422507 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.422483    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwfzc\" (UniqueName: \"kubernetes.io/projected/95ba72ee-301b-4265-b57a-013c9f16d8b5-kube-api-access-lwfzc\") pod \"dns-operator-controller-manager-65bfd79598-9c76g\" (UID: \"95ba72ee-301b-4265-b57a-013c9f16d8b5\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g"
Oct 30 19:24:33.481525 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.481496    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwfzc\" (UniqueName: \"kubernetes.io/projected/95ba72ee-301b-4265-b57a-013c9f16d8b5-kube-api-access-lwfzc\") pod \"dns-operator-controller-manager-65bfd79598-9c76g\" (UID: \"95ba72ee-301b-4265-b57a-013c9f16d8b5\") " pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g"
Oct 30 19:24:33.619034 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.618950    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g"
Oct 30 19:24:33.992508 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:33.992480    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g"]
Oct 30 19:24:33.995925 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:24:33.995902    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ba72ee_301b_4265_b57a_013c9f16d8b5.slice/crio-b4b29e18020b4ab3d790512c46deb40066f49d5750e998629a0b40a627d722b3 WatchSource:0}: Error finding container b4b29e18020b4ab3d790512c46deb40066f49d5750e998629a0b40a627d722b3: Status 404 returned error can't find the container with id b4b29e18020b4ab3d790512c46deb40066f49d5750e998629a0b40a627d722b3
Oct 30 19:24:34.946626 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:34.946597    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g" event={"ID":"95ba72ee-301b-4265-b57a-013c9f16d8b5","Type":"ContainerStarted","Data":"b4b29e18020b4ab3d790512c46deb40066f49d5750e998629a0b40a627d722b3"}
Oct 30 19:24:35.196070 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:24:35.196030    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:24:36.415057 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:36.415031    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-2hxh9"]
Oct 30 19:24:36.418458 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:36.418414    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-2hxh9"
Oct 30 19:24:36.446213 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:36.446188    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4slwt\" (UniqueName: \"kubernetes.io/projected/abce33e0-e121-4270-9869-6ffd925ec08c-kube-api-access-4slwt\") pod \"authorino-operator-68d6dd76dc-2hxh9\" (UID: \"abce33e0-e121-4270-9869-6ffd925ec08c\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-2hxh9"
Oct 30 19:24:36.519837 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:36.519769    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hcvfl\""
Oct 30 19:24:36.522094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:36.522024    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-2hxh9"]
Oct 30 19:24:36.546933 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:36.546906    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4slwt\" (UniqueName: \"kubernetes.io/projected/abce33e0-e121-4270-9869-6ffd925ec08c-kube-api-access-4slwt\") pod \"authorino-operator-68d6dd76dc-2hxh9\" (UID: \"abce33e0-e121-4270-9869-6ffd925ec08c\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-2hxh9"
Oct 30 19:24:36.612317 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:36.612286    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4slwt\" (UniqueName: \"kubernetes.io/projected/abce33e0-e121-4270-9869-6ffd925ec08c-kube-api-access-4slwt\") pod \"authorino-operator-68d6dd76dc-2hxh9\" (UID: \"abce33e0-e121-4270-9869-6ffd925ec08c\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-2hxh9"
Oct 30 19:24:36.734379 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:36.734306    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-2hxh9"
Oct 30 19:24:37.114436 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:37.113178    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-2hxh9"]
Oct 30 19:24:37.961129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:37.961103    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g" event={"ID":"95ba72ee-301b-4265-b57a-013c9f16d8b5","Type":"ContainerStarted","Data":"855da81db2fe38b6e87a06fb90d0695d4ea390ac0bbda7403b39d1c081f2092b"}
Oct 30 19:24:37.961500 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:37.961180    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g"
Oct 30 19:24:37.962477 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:37.962457    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-2hxh9" event={"ID":"abce33e0-e121-4270-9869-6ffd925ec08c","Type":"ContainerStarted","Data":"86616400ce507e7f67f2f7a67db0bb0abea2979e34abe2ba4fea4ad46b1a53e5"}
Oct 30 19:24:38.120072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:38.120024    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g" podStartSLOduration=1.227571206 podStartE2EDuration="5.120011289s" totalImagesPullingTime="3.892440083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:38.118799308 +0000 UTC m=+525.330016471" watchObservedRunningTime="2026-10-30 19:24:38.120011289 +0000 UTC m=+525.331228437"
Oct 30 19:24:40.975964 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:40.975925    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-2hxh9" event={"ID":"abce33e0-e121-4270-9869-6ffd925ec08c","Type":"ContainerStarted","Data":"60eaf05beb78659496066e2802d0718bba661d4bfa2e520367b0132ee23c378b"}
Oct 30 19:24:40.976346 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:40.976080    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-2hxh9"
Oct 30 19:24:41.093418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:41.093365    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-2hxh9" podStartSLOduration=1.605557015 podStartE2EDuration="5.093352479s" totalImagesPullingTime="3.487795464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:41.091410815 +0000 UTC m=+528.302627960" watchObservedRunningTime="2026-10-30 19:24:41.093352479 +0000 UTC m=+528.304569685"
Oct 30 19:24:48.968570 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:48.968537    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65bfd79598-9c76g"
Oct 30 19:24:51.982067 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:24:51.982036    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-2hxh9"
Oct 30 19:25:05.195512 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:25:05.195479    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:25:35.194547 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:25:35.194512    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:26:02.894758 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:02.894731    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"]
Oct 30 19:26:02.898089 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:02.898071    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:02.952415 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:02.952391    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8l6kf\""
Oct 30 19:26:02.997994 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:02.997969    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhv7f\" (UniqueName: \"kubernetes.io/projected/fdaa8038-e202-457e-9393-e7b17a3b9279-kube-api-access-qhv7f\") pod \"limitador-limitador-74dfcfb894-pk7v7\" (UID: \"fdaa8038-e202-457e-9393-e7b17a3b9279\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:02.998114 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:02.997997    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fdaa8038-e202-457e-9393-e7b17a3b9279-config-file\") pod \"limitador-limitador-74dfcfb894-pk7v7\" (UID: \"fdaa8038-e202-457e-9393-e7b17a3b9279\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:03.001463 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:03.001441    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 19:26:03.003210 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:03.003188    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"]
Oct 30 19:26:03.098473 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:03.098446    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhv7f\" (UniqueName: \"kubernetes.io/projected/fdaa8038-e202-457e-9393-e7b17a3b9279-kube-api-access-qhv7f\") pod \"limitador-limitador-74dfcfb894-pk7v7\" (UID: \"fdaa8038-e202-457e-9393-e7b17a3b9279\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:03.098586 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:03.098477    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fdaa8038-e202-457e-9393-e7b17a3b9279-config-file\") pod \"limitador-limitador-74dfcfb894-pk7v7\" (UID: \"fdaa8038-e202-457e-9393-e7b17a3b9279\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:03.099184 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:03.099162    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fdaa8038-e202-457e-9393-e7b17a3b9279-config-file\") pod \"limitador-limitador-74dfcfb894-pk7v7\" (UID: \"fdaa8038-e202-457e-9393-e7b17a3b9279\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:03.139018 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:03.138992    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"]
Oct 30 19:26:03.155312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:03.155256    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhv7f\" (UniqueName: \"kubernetes.io/projected/fdaa8038-e202-457e-9393-e7b17a3b9279-kube-api-access-qhv7f\") pod \"limitador-limitador-74dfcfb894-pk7v7\" (UID: \"fdaa8038-e202-457e-9393-e7b17a3b9279\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:03.210116 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:03.210092    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:03.577348 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:03.577191    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"]
Oct 30 19:26:03.580150 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:26:03.580120    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdaa8038_e202_457e_9393_e7b17a3b9279.slice/crio-3d8431aa5526e9ffcfeebd749448c8c5c561c0664cbb467f181961042e5302d3 WatchSource:0}: Error finding container 3d8431aa5526e9ffcfeebd749448c8c5c561c0664cbb467f181961042e5302d3: Status 404 returned error can't find the container with id 3d8431aa5526e9ffcfeebd749448c8c5c561c0664cbb467f181961042e5302d3
Oct 30 19:26:04.321241 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:04.321210    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7" event={"ID":"fdaa8038-e202-457e-9393-e7b17a3b9279","Type":"ContainerStarted","Data":"3d8431aa5526e9ffcfeebd749448c8c5c561c0664cbb467f181961042e5302d3"}
Oct 30 19:26:05.194588 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:26:05.194554    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:26:06.330453 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:06.330425    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7" event={"ID":"fdaa8038-e202-457e-9393-e7b17a3b9279","Type":"ContainerStarted","Data":"ebff75f5a05c392614756284b99c62d0ce2443068b051b8d49d7fa0a33ba7e8f"}
Oct 30 19:26:06.330787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:06.330523    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:06.444984 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:06.444940    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7" podStartSLOduration=1.8222473479999999 podStartE2EDuration="4.44492557s" totalImagesPullingTime="2.622678222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:06.444194447 +0000 UTC m=+613.655411617" watchObservedRunningTime="2026-10-30 19:26:06.44492557 +0000 UTC m=+613.656142717"
Oct 30 19:26:17.335468 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:17.335443    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:21.521902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:21.521872    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"]
Oct 30 19:26:21.522358 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:21.522039    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7" podUID="fdaa8038-e202-457e-9393-e7b17a3b9279" containerName="limitador" containerID="cri-o://ebff75f5a05c392614756284b99c62d0ce2443068b051b8d49d7fa0a33ba7e8f" gracePeriod=30
Oct 30 19:26:22.406770 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:22.406734    4420 generic.go:356] "Generic (PLEG): container finished" podID="fdaa8038-e202-457e-9393-e7b17a3b9279" containerID="ebff75f5a05c392614756284b99c62d0ce2443068b051b8d49d7fa0a33ba7e8f" exitCode=0
Oct 30 19:26:22.406959 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:22.406860    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7" event={"ID":"fdaa8038-e202-457e-9393-e7b17a3b9279","Type":"ContainerDied","Data":"ebff75f5a05c392614756284b99c62d0ce2443068b051b8d49d7fa0a33ba7e8f"}
Oct 30 19:26:22.454940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:22.454918    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:22.545731 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:22.545711    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdaa8038-e202-457e-9393-e7b17a3b9279-config-file\" (UniqueName: \"kubernetes.io/configmap/fdaa8038-e202-457e-9393-e7b17a3b9279-config-file\") pod \"fdaa8038-e202-457e-9393-e7b17a3b9279\" (UID: \"fdaa8038-e202-457e-9393-e7b17a3b9279\") "
Oct 30 19:26:22.546061 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:22.545868    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdaa8038-e202-457e-9393-e7b17a3b9279-kube-api-access-qhv7f\" (UniqueName: \"kubernetes.io/projected/fdaa8038-e202-457e-9393-e7b17a3b9279-kube-api-access-qhv7f\") pod \"fdaa8038-e202-457e-9393-e7b17a3b9279\" (UID: \"fdaa8038-e202-457e-9393-e7b17a3b9279\") "
Oct 30 19:26:22.546117 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:22.546093    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdaa8038-e202-457e-9393-e7b17a3b9279-config-file" pod "fdaa8038-e202-457e-9393-e7b17a3b9279" (UID: "fdaa8038-e202-457e-9393-e7b17a3b9279"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:22.548000 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:22.547976    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdaa8038-e202-457e-9393-e7b17a3b9279-kube-api-access-qhv7f" pod "fdaa8038-e202-457e-9393-e7b17a3b9279" (UID: "fdaa8038-e202-457e-9393-e7b17a3b9279"). InnerVolumeSpecName "kube-api-access-qhv7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:22.646509 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:22.646489    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhv7f\" (UniqueName: \"kubernetes.io/projected/fdaa8038-e202-457e-9393-e7b17a3b9279-kube-api-access-qhv7f\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:26:22.646622 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:22.646513    4420 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fdaa8038-e202-457e-9393-e7b17a3b9279-config-file\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:26:23.411449 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:23.411420    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"
Oct 30 19:26:23.411608 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:23.411419    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-pk7v7" event={"ID":"fdaa8038-e202-457e-9393-e7b17a3b9279","Type":"ContainerDied","Data":"3d8431aa5526e9ffcfeebd749448c8c5c561c0664cbb467f181961042e5302d3"}
Oct 30 19:26:23.411608 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:23.411551    4420 scope.go:122] "RemoveContainer" containerID="ebff75f5a05c392614756284b99c62d0ce2443068b051b8d49d7fa0a33ba7e8f"
Oct 30 19:26:23.573607 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:23.573575    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"]
Oct 30 19:26:23.622790 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:23.622763    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-pk7v7"]
Oct 30 19:26:24.894012 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:24.893985    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdaa8038-e202-457e-9393-e7b17a3b9279" path="/var/lib/kubelet/pods/fdaa8038-e202-457e-9393-e7b17a3b9279/volumes"
Oct 30 19:26:35.195359 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:26:35.195321    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:26:38.677196 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.677162    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"]
Oct 30 19:26:38.677564 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.677502    4420 cpu_manager.go:414] "containerMap: removing container" podUID="fdaa8038-e202-457e-9393-e7b17a3b9279" containerName="limitador"
Oct 30 19:26:38.677564 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.677513    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdaa8038-e202-457e-9393-e7b17a3b9279" containerName="limitador"
Oct 30 19:26:38.677624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.677579    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdaa8038-e202-457e-9393-e7b17a3b9279" containerName="limitador"
Oct 30 19:26:38.680398 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.680382    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"
Oct 30 19:26:38.732615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.732587    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 19:26:38.732810 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.732664    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 19:26:38.767094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.767063    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7022b85c-4fdf-4e78-bfda-a7d9ce4a0572-data\") pod \"postgres-7cdbf6dc44-h9nq6\" (UID: \"7022b85c-4fdf-4e78-bfda-a7d9ce4a0572\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"
Oct 30 19:26:38.767222 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.767132    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5ksp\" (UniqueName: \"kubernetes.io/projected/7022b85c-4fdf-4e78-bfda-a7d9ce4a0572-kube-api-access-f5ksp\") pod \"postgres-7cdbf6dc44-h9nq6\" (UID: \"7022b85c-4fdf-4e78-bfda-a7d9ce4a0572\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"
Oct 30 19:26:38.781796 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.781774    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 19:26:38.781923 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.781854    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-d22jk\""
Oct 30 19:26:38.783160 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.783142    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"]
Oct 30 19:26:38.868214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.868187    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7022b85c-4fdf-4e78-bfda-a7d9ce4a0572-data\") pod \"postgres-7cdbf6dc44-h9nq6\" (UID: \"7022b85c-4fdf-4e78-bfda-a7d9ce4a0572\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"
Oct 30 19:26:38.868350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.868257    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5ksp\" (UniqueName: \"kubernetes.io/projected/7022b85c-4fdf-4e78-bfda-a7d9ce4a0572-kube-api-access-f5ksp\") pod \"postgres-7cdbf6dc44-h9nq6\" (UID: \"7022b85c-4fdf-4e78-bfda-a7d9ce4a0572\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"
Oct 30 19:26:38.868634 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.868615    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7022b85c-4fdf-4e78-bfda-a7d9ce4a0572-data\") pod \"postgres-7cdbf6dc44-h9nq6\" (UID: \"7022b85c-4fdf-4e78-bfda-a7d9ce4a0572\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"
Oct 30 19:26:38.924777 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.924754    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5ksp\" (UniqueName: \"kubernetes.io/projected/7022b85c-4fdf-4e78-bfda-a7d9ce4a0572-kube-api-access-f5ksp\") pod \"postgres-7cdbf6dc44-h9nq6\" (UID: \"7022b85c-4fdf-4e78-bfda-a7d9ce4a0572\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"
Oct 30 19:26:38.993632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:38.993585    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"
Oct 30 19:26:39.362890 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:39.362804    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"]
Oct 30 19:26:39.365734 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:26:39.365701    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7022b85c_4fdf_4e78_bfda_a7d9ce4a0572.slice/crio-83d0cdf44835843f6f25bbbf83082c00d4e303eda318692cdc394bed29d5d231 WatchSource:0}: Error finding container 83d0cdf44835843f6f25bbbf83082c00d4e303eda318692cdc394bed29d5d231: Status 404 returned error can't find the container with id 83d0cdf44835843f6f25bbbf83082c00d4e303eda318692cdc394bed29d5d231
Oct 30 19:26:39.367082 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:39.367063    4420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:26:39.473950 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:39.473915    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6" event={"ID":"7022b85c-4fdf-4e78-bfda-a7d9ce4a0572","Type":"ContainerStarted","Data":"83d0cdf44835843f6f25bbbf83082c00d4e303eda318692cdc394bed29d5d231"}
Oct 30 19:26:50.521495 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:50.521458    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6" event={"ID":"7022b85c-4fdf-4e78-bfda-a7d9ce4a0572","Type":"ContainerStarted","Data":"be1d1d07de9c0fdf835bc71ad1f15e3329464feac19e68498dc1cfb843873798"}
Oct 30 19:26:50.521893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:50.521558    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"
Oct 30 19:26:50.634995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:50.634950    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6" podStartSLOduration=1.8814094099999998 podStartE2EDuration="12.634938145s" totalImagesPullingTime="10.753528735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:26:50.633470339 +0000 UTC m=+657.844687485" watchObservedRunningTime="2026-10-30 19:26:50.634938145 +0000 UTC m=+657.846155290"
Oct 30 19:26:56.553307 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:26:56.553281    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-h9nq6"
Oct 30 19:27:05.195208 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:27:05.195126    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:27:18.464204 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.464172    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-577f596567-2q2l8"]
Oct 30 19:27:18.466687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.466668    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:18.479263 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.479237    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-webhook-cert\") pod \"maas-controller-577f596567-2q2l8\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") " pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:18.479356 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.479303    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-metrics-certs\") pod \"maas-controller-577f596567-2q2l8\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") " pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:18.479456 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.479363    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssn52\" (UniqueName: \"kubernetes.io/projected/3696ebfe-0f65-4ce0-9079-2a43379e92c4-kube-api-access-ssn52\") pod \"maas-controller-577f596567-2q2l8\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") " pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:18.522958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.522935    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 19:27:18.522958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.522958    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 19:27:18.523203 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.523187    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sq2hw\""
Oct 30 19:27:18.523270 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.523225    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 19:27:18.529241 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.529223    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-2q2l8"]
Oct 30 19:27:18.579895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.579870    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-metrics-certs\") pod \"maas-controller-577f596567-2q2l8\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") " pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:18.580013 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.579899    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssn52\" (UniqueName: \"kubernetes.io/projected/3696ebfe-0f65-4ce0-9079-2a43379e92c4-kube-api-access-ssn52\") pod \"maas-controller-577f596567-2q2l8\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") " pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:18.580013 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.579926    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-webhook-cert\") pod \"maas-controller-577f596567-2q2l8\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") " pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:18.582708 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.582669    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-webhook-cert\") pod \"maas-controller-577f596567-2q2l8\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") " pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:18.582931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.582913    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-metrics-certs\") pod \"maas-controller-577f596567-2q2l8\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") " pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:18.636940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.636912    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssn52\" (UniqueName: \"kubernetes.io/projected/3696ebfe-0f65-4ce0-9079-2a43379e92c4-kube-api-access-ssn52\") pod \"maas-controller-577f596567-2q2l8\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") " pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:18.777965 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.777883    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:18.883584 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.883558    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-fkfr5"]
Oct 30 19:27:18.886316 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.886298    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:18.946691 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.946668    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-fkfr5"]
Oct 30 19:27:18.990833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.990788    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-metrics-certs\") pod \"maas-controller-6b5d5b686b-fkfr5\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") " pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:18.990970 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.990842    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-webhook-cert\") pod \"maas-controller-6b5d5b686b-fkfr5\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") " pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:18.990970 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:18.990901    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8l45\" (UniqueName: \"kubernetes.io/projected/6981f9c5-0fa0-4050-8b66-62b07f42ab85-kube-api-access-s8l45\") pod \"maas-controller-6b5d5b686b-fkfr5\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") " pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:19.092130 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:19.092063    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-webhook-cert\") pod \"maas-controller-6b5d5b686b-fkfr5\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") " pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:19.092130 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:19.092091    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8l45\" (UniqueName: \"kubernetes.io/projected/6981f9c5-0fa0-4050-8b66-62b07f42ab85-kube-api-access-s8l45\") pod \"maas-controller-6b5d5b686b-fkfr5\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") " pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:19.092317 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:19.092161    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-metrics-certs\") pod \"maas-controller-6b5d5b686b-fkfr5\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") " pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:19.094592 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:19.094570    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-webhook-cert\") pod \"maas-controller-6b5d5b686b-fkfr5\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") " pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:19.094686 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:19.094620    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-metrics-certs\") pod \"maas-controller-6b5d5b686b-fkfr5\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") " pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:19.148280 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:19.148263    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8l45\" (UniqueName: \"kubernetes.io/projected/6981f9c5-0fa0-4050-8b66-62b07f42ab85-kube-api-access-s8l45\") pod \"maas-controller-6b5d5b686b-fkfr5\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") " pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:19.154359 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:19.154299    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-577f596567-2q2l8"]
Oct 30 19:27:19.156999 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:27:19.156974    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3696ebfe_0f65_4ce0_9079_2a43379e92c4.slice/crio-7a1bde9a45b096bd2e214156e9296df1e24608569af86076edaed18ca14ba2a8 WatchSource:0}: Error finding container 7a1bde9a45b096bd2e214156e9296df1e24608569af86076edaed18ca14ba2a8: Status 404 returned error can't find the container with id 7a1bde9a45b096bd2e214156e9296df1e24608569af86076edaed18ca14ba2a8
Oct 30 19:27:19.198871 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:19.198848    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:19.565130 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:19.565058    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-fkfr5"]
Oct 30 19:27:19.568080 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:27:19.568054    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6981f9c5_0fa0_4050_8b66_62b07f42ab85.slice/crio-500d137378eb453b37d0a295584cb9f1ae4ebc7cce263d027cbe4f934f56bd35 WatchSource:0}: Error finding container 500d137378eb453b37d0a295584cb9f1ae4ebc7cce263d027cbe4f934f56bd35: Status 404 returned error can't find the container with id 500d137378eb453b37d0a295584cb9f1ae4ebc7cce263d027cbe4f934f56bd35
Oct 30 19:27:19.642763 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:19.642739    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5" event={"ID":"6981f9c5-0fa0-4050-8b66-62b07f42ab85","Type":"ContainerStarted","Data":"500d137378eb453b37d0a295584cb9f1ae4ebc7cce263d027cbe4f934f56bd35"}
Oct 30 19:27:19.643908 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:19.643880    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2q2l8" event={"ID":"3696ebfe-0f65-4ce0-9079-2a43379e92c4","Type":"ContainerStarted","Data":"7a1bde9a45b096bd2e214156e9296df1e24608569af86076edaed18ca14ba2a8"}
Oct 30 19:27:22.664190 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:22.664150    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2q2l8" event={"ID":"3696ebfe-0f65-4ce0-9079-2a43379e92c4","Type":"ContainerStarted","Data":"dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b"}
Oct 30 19:27:22.664968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:22.664932    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:22.819248 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:22.819207    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-577f596567-2q2l8" podStartSLOduration=1.438920753 podStartE2EDuration="4.819192923s" totalImagesPullingTime="3.38027217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:22.817675831 +0000 UTC m=+690.028892986" watchObservedRunningTime="2026-10-30 19:27:22.819192923 +0000 UTC m=+690.030410089"
Oct 30 19:27:23.669686 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:23.669645    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5" event={"ID":"6981f9c5-0fa0-4050-8b66-62b07f42ab85","Type":"ContainerStarted","Data":"936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27"}
Oct 30 19:27:23.670279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:23.669838    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:23.786987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:23.786900    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5" podStartSLOduration=2.577954187 podStartE2EDuration="5.786879852s" totalImagesPullingTime="3.208925665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:23.786338775 +0000 UTC m=+690.997555920" watchObservedRunningTime="2026-10-30 19:27:23.786879852 +0000 UTC m=+690.998096999"
Oct 30 19:27:27.506960 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.506931    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"]
Oct 30 19:27:27.510632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.510608    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:27.562608 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.562586    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 19:27:27.562747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.562728    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 19:27:27.562856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.562837    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 19:27:27.566160 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.566142    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:27.566218 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.566172    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:27.566298 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.566280    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mw6\" (UniqueName: \"kubernetes.io/projected/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-kube-api-access-s8mw6\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:27.566333 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.566314    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:27.612183 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.612153    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-96tsf\""
Oct 30 19:27:27.613713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.613694    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"]
Oct 30 19:27:27.667435 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.667408    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:27.667566 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.667455    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:27.667566 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:27:27.667523    4420 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 30 19:27:27.667566 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:27:27.667544    4420 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 19:27:27.667668 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.667562    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:27.667668 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:27:27.667596    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-proxy-source podName:5f627072-bcb8-4998-8cb0-51c3e4ead9ac nodeName:}" failed. No retries permitted until 2026-10-30 19:27:28.167579597 +0000 UTC m=+695.378796750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-proxy-source") pod "usage-logs-tenancy-proxy-7d94f55f4b-s4vzh" (UID: "5f627072-bcb8-4998-8cb0-51c3e4ead9ac") : configmap "usage-tenancy-proxy-source" not found
Oct 30 19:27:27.667668 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:27:27.667620    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-tls-cert podName:5f627072-bcb8-4998-8cb0-51c3e4ead9ac nodeName:}" failed. No retries permitted until 2026-10-30 19:27:28.167614445 +0000 UTC m=+695.378831568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-tls-cert") pod "usage-logs-tenancy-proxy-7d94f55f4b-s4vzh" (UID: "5f627072-bcb8-4998-8cb0-51c3e4ead9ac") : secret "usage-logs-tenancy-proxy-tls" not found
Oct 30 19:27:27.667790 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.667691    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mw6\" (UniqueName: \"kubernetes.io/projected/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-kube-api-access-s8mw6\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:27.668097 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.668081    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-service-ca\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:27.727537 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:27.727515    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mw6\" (UniqueName: \"kubernetes.io/projected/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-kube-api-access-s8mw6\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:28.172863 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:28.172835    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:28.173058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:28.172923    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:28.173574 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:28.173551    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-proxy-source\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:28.175246 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:28.175226    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f627072-bcb8-4998-8cb0-51c3e4ead9ac-tls-cert\") pod \"usage-logs-tenancy-proxy-7d94f55f4b-s4vzh\" (UID: \"5f627072-bcb8-4998-8cb0-51c3e4ead9ac\") " pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:28.423118 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:28.423049    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:28.799467 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:28.799443    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"]
Oct 30 19:27:28.801277 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:27:28.801253    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f627072_bcb8_4998_8cb0_51c3e4ead9ac.slice/crio-d8bdaec717d79c7a3b3f8426867ec71bc8f3be7ef397c1e517304d2c99c7286e WatchSource:0}: Error finding container d8bdaec717d79c7a3b3f8426867ec71bc8f3be7ef397c1e517304d2c99c7286e: Status 404 returned error can't find the container with id d8bdaec717d79c7a3b3f8426867ec71bc8f3be7ef397c1e517304d2c99c7286e
Oct 30 19:27:29.695921 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:29.695885    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh" event={"ID":"5f627072-bcb8-4998-8cb0-51c3e4ead9ac","Type":"ContainerStarted","Data":"d8bdaec717d79c7a3b3f8426867ec71bc8f3be7ef397c1e517304d2c99c7286e"}
Oct 30 19:27:34.675990 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:34.675951    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:34.676483 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:34.676040    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 19:27:34.843976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:34.843543    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-2q2l8"]
Oct 30 19:27:34.843976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:34.843772    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-577f596567-2q2l8" podUID="3696ebfe-0f65-4ce0-9079-2a43379e92c4" containerName="manager" containerID="cri-o://dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b" gracePeriod=30
Oct 30 19:27:35.126794 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.126768    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:35.204397 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:27:35.204352    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:27:35.241423 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.241351    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-metrics-certs\") pod \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") "
Oct 30 19:27:35.241423 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.241407    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3696ebfe-0f65-4ce0-9079-2a43379e92c4-kube-api-access-ssn52\" (UniqueName: \"kubernetes.io/projected/3696ebfe-0f65-4ce0-9079-2a43379e92c4-kube-api-access-ssn52\") pod \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") "
Oct 30 19:27:35.241619 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.241511    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-webhook-cert\") pod \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\" (UID: \"3696ebfe-0f65-4ce0-9079-2a43379e92c4\") "
Oct 30 19:27:35.244701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.244667    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-metrics-certs" pod "3696ebfe-0f65-4ce0-9079-2a43379e92c4" (UID: "3696ebfe-0f65-4ce0-9079-2a43379e92c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:35.250394 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.250364    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3696ebfe-0f65-4ce0-9079-2a43379e92c4-kube-api-access-ssn52" pod "3696ebfe-0f65-4ce0-9079-2a43379e92c4" (UID: "3696ebfe-0f65-4ce0-9079-2a43379e92c4"). InnerVolumeSpecName "kube-api-access-ssn52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:35.253305 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.253278    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-webhook-cert" pod "3696ebfe-0f65-4ce0-9079-2a43379e92c4" (UID: "3696ebfe-0f65-4ce0-9079-2a43379e92c4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:35.342381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.342350    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssn52\" (UniqueName: \"kubernetes.io/projected/3696ebfe-0f65-4ce0-9079-2a43379e92c4-kube-api-access-ssn52\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:27:35.342381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.342383    4420 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-webhook-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:27:35.342612 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.342395    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3696ebfe-0f65-4ce0-9079-2a43379e92c4-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:27:35.727953 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.727919    4420 generic.go:356] "Generic (PLEG): container finished" podID="3696ebfe-0f65-4ce0-9079-2a43379e92c4" containerID="dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b" exitCode=0
Oct 30 19:27:35.728535 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.727988    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-577f596567-2q2l8"
Oct 30 19:27:35.728535 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.728052    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2q2l8" event={"ID":"3696ebfe-0f65-4ce0-9079-2a43379e92c4","Type":"ContainerDied","Data":"dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b"}
Oct 30 19:27:35.728535 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.728099    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-577f596567-2q2l8" event={"ID":"3696ebfe-0f65-4ce0-9079-2a43379e92c4","Type":"ContainerDied","Data":"7a1bde9a45b096bd2e214156e9296df1e24608569af86076edaed18ca14ba2a8"}
Oct 30 19:27:35.728535 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.728127    4420 scope.go:122] "RemoveContainer" containerID="dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b"
Oct 30 19:27:35.738220 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.738200    4420 scope.go:122] "RemoveContainer" containerID="dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b"
Oct 30 19:27:35.738540 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:27:35.738476    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b\": container with ID starting with dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b not found: ID does not exist" containerID="dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b"
Oct 30 19:27:35.738540 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.738514    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b"} err="failed to get container status \"dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b\": rpc error: code = NotFound desc = could not find container \"dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b\": container with ID starting with dee99f4d7a9a1116f215ec144170ae456662ea22c1355b4dea80f98fb12c999b not found: ID does not exist"
Oct 30 19:27:35.892371 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.892335    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-577f596567-2q2l8"]
Oct 30 19:27:35.897546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:35.897522    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-577f596567-2q2l8"]
Oct 30 19:27:36.894222 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:36.894190    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3696ebfe-0f65-4ce0-9079-2a43379e92c4" path="/var/lib/kubelet/pods/3696ebfe-0f65-4ce0-9079-2a43379e92c4/volumes"
Oct 30 19:27:41.759740 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:41.759700    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh" event={"ID":"5f627072-bcb8-4998-8cb0-51c3e4ead9ac","Type":"ContainerStarted","Data":"f4ef74712c4ffdca463284f96a2232a2266ca78048c6bf12565ec81e1bda76a1"}
Oct 30 19:27:41.760120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:41.759884    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:27:41.873262 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:41.873209    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh" podStartSLOduration=2.958772182 podStartE2EDuration="14.873197807s" totalImagesPullingTime="11.914425625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:41.872529019 +0000 UTC m=+709.083746164" watchObservedRunningTime="2026-10-30 19:27:41.873197807 +0000 UTC m=+709.084414948"
Oct 30 19:27:52.769521 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:27:52.769493    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-7d94f55f4b-s4vzh"
Oct 30 19:28:05.194638 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:28:05.194603    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:28:35.196295 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:28:35.196214    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:29:05.195983 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:29:05.195948    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:29:21.178330 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.178295    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"]
Oct 30 19:29:21.179270 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.178896    4420 cpu_manager.go:414] "containerMap: removing container" podUID="3696ebfe-0f65-4ce0-9079-2a43379e92c4" containerName="manager"
Oct 30 19:29:21.179270 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.178914    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3696ebfe-0f65-4ce0-9079-2a43379e92c4" containerName="manager"
Oct 30 19:29:21.179270 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.178992    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3696ebfe-0f65-4ce0-9079-2a43379e92c4" containerName="manager"
Oct 30 19:29:21.181012 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.180993    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.233136 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.233113    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 19:29:21.233136 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.233132    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 19:29:21.252452 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.252430    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.252572 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.252455    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.252572 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.252495    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.252656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.252619    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d4afd-3115-469c-aa3f-ed96a8233238-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.252656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.252649    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkwdm\" (UniqueName: \"kubernetes.io/projected/958d4afd-3115-469c-aa3f-ed96a8233238-kube-api-access-lkwdm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.252731 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.252669    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.282484 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.282465    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2svj8\""
Oct 30 19:29:21.282593 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.282544    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 19:29:21.284144 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.284127    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"]
Oct 30 19:29:21.353868 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.353838    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.353868 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.353868    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.354021 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.353910    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.354021 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.353962    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d4afd-3115-469c-aa3f-ed96a8233238-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.354021 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.353984    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkwdm\" (UniqueName: \"kubernetes.io/projected/958d4afd-3115-469c-aa3f-ed96a8233238-kube-api-access-lkwdm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.354021 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.354014    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.354287 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.354267    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.354351 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.354303    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.354417 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.354400    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.356246 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.356226    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/958d4afd-3115-469c-aa3f-ed96a8233238-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.356496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.356480    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/958d4afd-3115-469c-aa3f-ed96a8233238-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.412278 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.412256    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkwdm\" (UniqueName: \"kubernetes.io/projected/958d4afd-3115-469c-aa3f-ed96a8233238-kube-api-access-lkwdm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4\" (UID: \"958d4afd-3115-469c-aa3f-ed96a8233238\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.492510 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.492451    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:21.866461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:21.866439    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"]
Oct 30 19:29:21.869045 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:29:21.869016    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod958d4afd_3115_469c_aa3f_ed96a8233238.slice/crio-91414a711aa21fad21963bf833119b4f6d6c1fba6cace42f13c8aaacb5f0f917 WatchSource:0}: Error finding container 91414a711aa21fad21963bf833119b4f6d6c1fba6cace42f13c8aaacb5f0f917: Status 404 returned error can't find the container with id 91414a711aa21fad21963bf833119b4f6d6c1fba6cace42f13c8aaacb5f0f917
Oct 30 19:29:22.175617 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:22.175588    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4" event={"ID":"958d4afd-3115-469c-aa3f-ed96a8233238","Type":"ContainerStarted","Data":"91414a711aa21fad21963bf833119b4f6d6c1fba6cace42f13c8aaacb5f0f917"}
Oct 30 19:29:24.186390 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:24.186357    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4" event={"ID":"958d4afd-3115-469c-aa3f-ed96a8233238","Type":"ContainerStarted","Data":"0d56372c56615621e0ea4403f5b73059354f68ba2a470447b74f514eb718a72e"}
Oct 30 19:29:24.301041 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:24.300998    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4" podStartSLOduration=1.4583200170000001 podStartE2EDuration="3.300983432s" totalImagesPullingTime="1.842663415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:24.300655733 +0000 UTC m=+811.511872876" watchObservedRunningTime="2026-10-30 19:29:24.300983432 +0000 UTC m=+811.512200577"
Oct 30 19:29:24.492613 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:24.492556    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:29.201615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.201586    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"]
Oct 30 19:29:29.204101 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.204065    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.296327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.296297    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 19:29:29.313640 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.313609    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ljcv\" (UniqueName: \"kubernetes.io/projected/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-kube-api-access-7ljcv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.313760 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.313652    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.313760 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.313679    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.313876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.313781    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.313876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.313808    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.313965 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.313946    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.317014 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.316999    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"]
Oct 30 19:29:29.414724 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.414692    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljcv\" (UniqueName: \"kubernetes.io/projected/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-kube-api-access-7ljcv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.414893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.414734    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.414893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.414759    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.414893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.414837    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.414893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.414868    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.415115 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.414920    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.415182 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.415164    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.415182 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.415176    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.415338 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.415242    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.417141 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.417121    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.417384 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.417366    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.473111 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.473057    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljcv\" (UniqueName: \"kubernetes.io/projected/bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4-kube-api-access-7ljcv\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7pz2b\" (UID: \"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.515902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.515877    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:29.895174 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:29.895146    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"]
Oct 30 19:29:29.899248 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:29:29.899225    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd70a6e_d10d_43bf_9b27_1d9b7e7334f4.slice/crio-c1f504bc5f672dec039b08f318a9bacfa811058a166eb2b777f2ffbeaa58a85b WatchSource:0}: Error finding container c1f504bc5f672dec039b08f318a9bacfa811058a166eb2b777f2ffbeaa58a85b: Status 404 returned error can't find the container with id c1f504bc5f672dec039b08f318a9bacfa811058a166eb2b777f2ffbeaa58a85b
Oct 30 19:29:30.212406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:30.212335    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b" event={"ID":"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4","Type":"ContainerStarted","Data":"c1f504bc5f672dec039b08f318a9bacfa811058a166eb2b777f2ffbeaa58a85b"}
Oct 30 19:29:31.217690 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:31.217658    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b" event={"ID":"bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4","Type":"ContainerStarted","Data":"694dbdab57648bed055eced399308e61396ce0452623ea5587b511739a8e189a"}
Oct 30 19:29:31.335727 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:31.335683    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b" podStartSLOduration=1.743053206 podStartE2EDuration="2.335669209s" totalImagesPullingTime="592.616003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:31.334677541 +0000 UTC m=+818.545894685" watchObservedRunningTime="2026-10-30 19:29:31.335669209 +0000 UTC m=+818.546886395"
Oct 30 19:29:31.493672 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:31.493609    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:31.505313 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:31.505296    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:31.516585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:31.516564    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:32.234893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:32.234868    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4"
Oct 30 19:29:35.195032 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:29:35.194998    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:29:35.762868 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.762842    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"]
Oct 30 19:29:35.765152 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.765138    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.866415 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.866390    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 30 19:29:35.867993 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.867974    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.868074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.868006    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.868074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.868031    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/593f49af-9c5d-4813-851f-2e1481f27476-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.868158 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.868136    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.868191 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.868182    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.868242 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.868230    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tx6j\" (UniqueName: \"kubernetes.io/projected/593f49af-9c5d-4813-851f-2e1481f27476-kube-api-access-6tx6j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.868532 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.868514    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"]
Oct 30 19:29:35.968701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.968667    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tx6j\" (UniqueName: \"kubernetes.io/projected/593f49af-9c5d-4813-851f-2e1481f27476-kube-api-access-6tx6j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.968849 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.968738    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.968849 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.968772    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.968945 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.968917    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/593f49af-9c5d-4813-851f-2e1481f27476-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.969000 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.968961    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.969000 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.968993    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.969192 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.969171    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.969282 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.969240    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.969346 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.969319    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.971237 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.971216    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/593f49af-9c5d-4813-851f-2e1481f27476-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:35.971483 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:35.971466    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/593f49af-9c5d-4813-851f-2e1481f27476-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:36.045538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:36.045478    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tx6j\" (UniqueName: \"kubernetes.io/projected/593f49af-9c5d-4813-851f-2e1481f27476-kube-api-access-6tx6j\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4\" (UID: \"593f49af-9c5d-4813-851f-2e1481f27476\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:36.076336 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:36.076310    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:36.458776 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:36.458749    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"]
Oct 30 19:29:36.462153 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:29:36.462123    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593f49af_9c5d_4813_851f_2e1481f27476.slice/crio-42d26b35dedc77469a72df6664aad82fb55a878a3de3b98889eef6d82f50467e WatchSource:0}: Error finding container 42d26b35dedc77469a72df6664aad82fb55a878a3de3b98889eef6d82f50467e: Status 404 returned error can't find the container with id 42d26b35dedc77469a72df6664aad82fb55a878a3de3b98889eef6d82f50467e
Oct 30 19:29:37.243938 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:37.243905    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4" event={"ID":"593f49af-9c5d-4813-851f-2e1481f27476","Type":"ContainerStarted","Data":"5bed907d8c43d8f89508d9ae7481c77c7473a0ce6318fb994e7b808f7b47e413"}
Oct 30 19:29:37.244103 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:37.243990    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4" event={"ID":"593f49af-9c5d-4813-851f-2e1481f27476","Type":"ContainerStarted","Data":"42d26b35dedc77469a72df6664aad82fb55a878a3de3b98889eef6d82f50467e"}
Oct 30 19:29:37.362767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:37.362679    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4" podStartSLOduration=1.763209085 podStartE2EDuration="2.3626656s" totalImagesPullingTime="599.456515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:37.361003182 +0000 UTC m=+824.572220328" watchObservedRunningTime="2026-10-30 19:29:37.3626656 +0000 UTC m=+824.573882745"
Oct 30 19:29:38.076830 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:38.076789    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:39.516579 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:39.516547    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:39.528498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:39.528472    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:40.267611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:40.267586    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7pz2b"
Oct 30 19:29:46.077058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.077026    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:46.088807 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.088785    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:46.295044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.295014    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4"
Oct 30 19:29:46.313633 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.313608    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"]
Oct 30 19:29:46.316214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.316201    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.369117 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.369049    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 30 19:29:46.377593 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.377567    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"]
Oct 30 19:29:46.448982 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.448956    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.449141 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.448997    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fcj5\" (UniqueName: \"kubernetes.io/projected/e13895f0-43af-4669-8830-40b9c80f2a9b-kube-api-access-8fcj5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.449141 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.449018    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13895f0-43af-4669-8830-40b9c80f2a9b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.449141 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.449125    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.449272 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.449151    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.449272 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.449184    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.550100 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.550063    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.550267 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.550128    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fcj5\" (UniqueName: \"kubernetes.io/projected/e13895f0-43af-4669-8830-40b9c80f2a9b-kube-api-access-8fcj5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.550267 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.550163    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13895f0-43af-4669-8830-40b9c80f2a9b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.550267 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.550233    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.550426 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.550268    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.550426 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.550335    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.550531 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.550477    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.550628 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.550608    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.550672 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.550646    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.552606 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.552590    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e13895f0-43af-4669-8830-40b9c80f2a9b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.552696 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.552677    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e13895f0-43af-4669-8830-40b9c80f2a9b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.607928 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.607906    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fcj5\" (UniqueName: \"kubernetes.io/projected/e13895f0-43af-4669-8830-40b9c80f2a9b-kube-api-access-8fcj5\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5\" (UID: \"e13895f0-43af-4669-8830-40b9c80f2a9b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:46.627536 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:46.627485    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:47.005389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:47.005354    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"]
Oct 30 19:29:47.014640 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:29:47.014616    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13895f0_43af_4669_8830_40b9c80f2a9b.slice/crio-56a404102698299959e05941f26ece13995591ac1f9b8148abb9825ff3d21852 WatchSource:0}: Error finding container 56a404102698299959e05941f26ece13995591ac1f9b8148abb9825ff3d21852: Status 404 returned error can't find the container with id 56a404102698299959e05941f26ece13995591ac1f9b8148abb9825ff3d21852
Oct 30 19:29:47.290487 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:47.290410    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5" event={"ID":"e13895f0-43af-4669-8830-40b9c80f2a9b","Type":"ContainerStarted","Data":"56a404102698299959e05941f26ece13995591ac1f9b8148abb9825ff3d21852"}
Oct 30 19:29:48.296479 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:48.296453    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5" event={"ID":"e13895f0-43af-4669-8830-40b9c80f2a9b","Type":"ContainerStarted","Data":"d2cfb2b6fc20cc7a47a270e50827c098659c640e802f6e2b753461113d4689e3"}
Oct 30 19:29:48.412056 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:48.412011    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5" podStartSLOduration=1.8056344640000002 podStartE2EDuration="2.411997208s" totalImagesPullingTime="606.362744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:48.410554103 +0000 UTC m=+835.621771246" watchObservedRunningTime="2026-10-30 19:29:48.411997208 +0000 UTC m=+835.623214353"
Oct 30 19:29:48.628638 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:48.628569    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:55.868436 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:55.868403    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"]
Oct 30 19:29:55.874157 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:55.874137    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:55.926421 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:55.926393    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f2npg\""
Oct 30 19:29:55.926421 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:55.926414    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:29:56.030285 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.030259    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.030285 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.030287    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.030464 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.030305    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.030464 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.030325    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rft5d\" (UniqueName: \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-kube-api-access-rft5d\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.030464 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.030393    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.030606 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.030572    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.030643 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.030630    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.030689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.030676    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.030790 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.030774    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.081307 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.081284    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"]
Oct 30 19:29:56.131739 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.131672    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rft5d\" (UniqueName: \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-kube-api-access-rft5d\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.131739 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.131711    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.131964 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.131766    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.131964 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.131791    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.132080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.131960    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.132080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.132004    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.132181 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.132098    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.132181 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.132127    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.132181 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.132152    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.132336 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.132201    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.132336 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.132234    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.132532 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.132503    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.132638 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.132511    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.132638 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.132604    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.134325 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.134296    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.134475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.134461    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.190120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.190100    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.190655 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.190638    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rft5d\" (UniqueName: \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-kube-api-access-rft5d\") pod \"maas-default-gateway-openshift-default-6db586dd6c-7kt6c\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.486783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.486757    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:29:56.628423 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.628392    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:56.640153 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.640127    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:29:56.922366 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:56.922342    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"]
Oct 30 19:29:56.924725 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:29:56.924693    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4ba9b8_54f5_4dab_9ca9_730994a59ebf.slice/crio-e089a1c19f50c4bca9dbc88c06c17ed53a442b0ea058727e3c9fe9c8bcd50064 WatchSource:0}: Error finding container e089a1c19f50c4bca9dbc88c06c17ed53a442b0ea058727e3c9fe9c8bcd50064: Status 404 returned error can't find the container with id e089a1c19f50c4bca9dbc88c06c17ed53a442b0ea058727e3c9fe9c8bcd50064
Oct 30 19:29:57.335797 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:57.335768    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" event={"ID":"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf","Type":"ContainerStarted","Data":"e089a1c19f50c4bca9dbc88c06c17ed53a442b0ea058727e3c9fe9c8bcd50064"}
Oct 30 19:29:57.346620 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:29:57.346598    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5"
Oct 30 19:30:00.172660 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.172626    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"]
Oct 30 19:30:00.179536 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.179510    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"
Oct 30 19:30:00.235756 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.235733    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"]
Oct 30 19:30:00.271560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.271527    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l624c\" (UniqueName: \"kubernetes.io/projected/633766ff-cc60-41bc-b92b-c0ba2d8159a0-kube-api-access-l624c\") pod \"maas-api-key-cleanup-29889810-7wzdb\" (UID: \"633766ff-cc60-41bc-b92b-c0ba2d8159a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"
Oct 30 19:30:00.271713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.271599    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/633766ff-cc60-41bc-b92b-c0ba2d8159a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-7wzdb\" (UID: \"633766ff-cc60-41bc-b92b-c0ba2d8159a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"
Oct 30 19:30:00.280995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.280965    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:30:00.372493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.372465    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l624c\" (UniqueName: \"kubernetes.io/projected/633766ff-cc60-41bc-b92b-c0ba2d8159a0-kube-api-access-l624c\") pod \"maas-api-key-cleanup-29889810-7wzdb\" (UID: \"633766ff-cc60-41bc-b92b-c0ba2d8159a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"
Oct 30 19:30:00.372756 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.372515    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/633766ff-cc60-41bc-b92b-c0ba2d8159a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-7wzdb\" (UID: \"633766ff-cc60-41bc-b92b-c0ba2d8159a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"
Oct 30 19:30:00.373157 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.373139    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/633766ff-cc60-41bc-b92b-c0ba2d8159a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889810-7wzdb\" (UID: \"633766ff-cc60-41bc-b92b-c0ba2d8159a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"
Oct 30 19:30:00.431170 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.431106    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l624c\" (UniqueName: \"kubernetes.io/projected/633766ff-cc60-41bc-b92b-c0ba2d8159a0-kube-api-access-l624c\") pod \"maas-api-key-cleanup-29889810-7wzdb\" (UID: \"633766ff-cc60-41bc-b92b-c0ba2d8159a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"
Oct 30 19:30:00.492710 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.492678    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"
Oct 30 19:30:00.869277 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.869251    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"]
Oct 30 19:30:00.872365 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:30:00.872330    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633766ff_cc60_41bc_b92b_c0ba2d8159a0.slice/crio-1daa25b90a7c7b75f9bd786e7ed05496a0e659a0edde05c0ab4215dd7c3d8316 WatchSource:0}: Error finding container 1daa25b90a7c7b75f9bd786e7ed05496a0e659a0edde05c0ab4215dd7c3d8316: Status 404 returned error can't find the container with id 1daa25b90a7c7b75f9bd786e7ed05496a0e659a0edde05c0ab4215dd7c3d8316
Oct 30 19:30:00.945866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.945832    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:00.945972 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.945914    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:00.945972 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:00.945958    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:01.357194 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:01.357164    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" event={"ID":"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf","Type":"ContainerStarted","Data":"897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c"}
Oct 30 19:30:01.358711 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:01.358683    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb" event={"ID":"633766ff-cc60-41bc-b92b-c0ba2d8159a0","Type":"ContainerStarted","Data":"1daa25b90a7c7b75f9bd786e7ed05496a0e659a0edde05c0ab4215dd7c3d8316"}
Oct 30 19:30:01.487059 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:01.487030    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:30:01.586471 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:01.586416    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" podStartSLOduration=2.567973046 podStartE2EDuration="6.586400253s" totalImagesPullingTime="4.018427207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:01.536706425 +0000 UTC m=+848.747923570" watchObservedRunningTime="2026-10-30 19:30:01.586400253 +0000 UTC m=+848.797617398"
Oct 30 19:30:02.488615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:02.488569    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7kt6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:30:02.489069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:02.488632    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" podUID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 19:30:03.487576 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:03.487539    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7kt6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:30:03.487756 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:03.487618    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" podUID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 19:30:04.487716 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:04.487629    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7kt6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:30:04.487716 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:04.487692    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" podUID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 19:30:05.213786 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:30:05.213750    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:30:05.487997 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:05.487904    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7kt6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:30:05.487997 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:05.487972    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" podUID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 19:30:06.408217 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:06.408183    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb" event={"ID":"633766ff-cc60-41bc-b92b-c0ba2d8159a0","Type":"ContainerStarted","Data":"a755eeddfc12a5f83ef3df309ab387c068a506bd4264b9eea7551d7bc64ad17f"}
Oct 30 19:30:06.488089 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:06.488056    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7kt6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:30:06.488473 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:06.488111    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" podUID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 19:30:06.541938 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:06.541895    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb" podStartSLOduration=1.6930321959999999 podStartE2EDuration="6.5418829s" totalImagesPullingTime="4.848850704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:06.540414241 +0000 UTC m=+853.751631389" watchObservedRunningTime="2026-10-30 19:30:06.5418829 +0000 UTC m=+853.753100045"
Oct 30 19:30:07.413727 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:07.413692    4420 generic.go:356] "Generic (PLEG): container finished" podID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerID="a755eeddfc12a5f83ef3df309ab387c068a506bd4264b9eea7551d7bc64ad17f" exitCode=60
Oct 30 19:30:07.413917 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:07.413782    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb" event={"ID":"633766ff-cc60-41bc-b92b-c0ba2d8159a0","Type":"ContainerDied","Data":"a755eeddfc12a5f83ef3df309ab387c068a506bd4264b9eea7551d7bc64ad17f"}
Oct 30 19:30:07.413995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:07.413982    4420 scope.go:122] "RemoveContainer" containerID="a755eeddfc12a5f83ef3df309ab387c068a506bd4264b9eea7551d7bc64ad17f"
Oct 30 19:30:07.487715 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:07.487688    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-7kt6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 30 19:30:07.487831 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:07.487756    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" podUID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 30 19:30:08.421251 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:08.421166    4420 generic.go:356] "Generic (PLEG): container finished" podID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerID="42b3f5668d4c3ce51e1aa120f0caf6d8a490a2e96bf4dea0e0d89278f10112ca" exitCode=60
Oct 30 19:30:08.421627 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:08.421259    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb" event={"ID":"633766ff-cc60-41bc-b92b-c0ba2d8159a0","Type":"ContainerDied","Data":"42b3f5668d4c3ce51e1aa120f0caf6d8a490a2e96bf4dea0e0d89278f10112ca"}
Oct 30 19:30:08.421627 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:08.421319    4420 scope.go:122] "RemoveContainer" containerID="a755eeddfc12a5f83ef3df309ab387c068a506bd4264b9eea7551d7bc64ad17f"
Oct 30 19:30:08.421627 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:08.421574    4420 scope.go:122] "RemoveContainer" containerID="42b3f5668d4c3ce51e1aa120f0caf6d8a490a2e96bf4dea0e0d89278f10112ca"
Oct 30 19:30:08.421890 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:30:08.421866    4420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-7wzdb_odh-ai-gateway-infra(633766ff-cc60-41bc-b92b-c0ba2d8159a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0"
Oct 30 19:30:09.034789 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:09.034765    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:30:09.035198 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:09.035177    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:30:09.035834 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:09.035799    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:30:09.427333 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:09.427310    4420 scope.go:122] "RemoveContainer" containerID="42b3f5668d4c3ce51e1aa120f0caf6d8a490a2e96bf4dea0e0d89278f10112ca"
Oct 30 19:30:09.427711 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:30:09.427544    4420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-7wzdb_odh-ai-gateway-infra(633766ff-cc60-41bc-b92b-c0ba2d8159a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0"
Oct 30 19:30:10.894553 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:10.894526    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"]
Oct 30 19:30:10.898968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:10.898953    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:10.973777 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:10.973751    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:10.973912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:10.973781    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:10.973912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:10.973805    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:10.973912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:10.973867    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d98efd-3b11-4464-bec1-5b46eec5d871-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:10.974026 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:10.973950    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:10.974026 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:10.973972    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:10.974026 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:10.974003    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx745\" (UniqueName: \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-kube-api-access-dx745\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:10.974026 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:10.974019    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:10.974148 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:10.974094    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.002444 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.002422    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"]
Oct 30 19:30:11.074603 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.074578    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.074734 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.074611    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.074838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.074801    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx745\" (UniqueName: \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-kube-api-access-dx745\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.074898 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.074843    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.074898 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.074860    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.074999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.074905    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.074999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.074925    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.074999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.074952    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.074999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.074973    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d98efd-3b11-4464-bec1-5b46eec5d871-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.075195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.075001    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.075294 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.075267    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.075428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.075277    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.075428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.075359    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.075650 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.075629    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d98efd-3b11-4464-bec1-5b46eec5d871-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.077046 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.077026    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.077237 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.077222    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.131837 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.131795    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.132008 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.131991    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx745\" (UniqueName: \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-kube-api-access-dx745\") pod \"maas-default-gateway-openshift-default-6db586dd6c-zhkqr\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.211166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.211111    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:11.633354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.633327    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"]
Oct 30 19:30:11.635496 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:30:11.635471    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d98efd_3b11_4464_bec1_5b46eec5d871.slice/crio-5cb2dc63aa0f1869daa91b918d13c882ebe2d7fca2eb7e3d5a3371312c2b6b56 WatchSource:0}: Error finding container 5cb2dc63aa0f1869daa91b918d13c882ebe2d7fca2eb7e3d5a3371312c2b6b56: Status 404 returned error can't find the container with id 5cb2dc63aa0f1869daa91b918d13c882ebe2d7fca2eb7e3d5a3371312c2b6b56
Oct 30 19:30:11.638592 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.638563    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:11.638666 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.638623    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:11.638666 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:11.638651    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:12.441831 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:12.441781    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" event={"ID":"a4d98efd-3b11-4464-bec1-5b46eec5d871","Type":"ContainerStarted","Data":"c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac"}
Oct 30 19:30:12.442190 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:12.441877    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" event={"ID":"a4d98efd-3b11-4464-bec1-5b46eec5d871","Type":"ContainerStarted","Data":"5cb2dc63aa0f1869daa91b918d13c882ebe2d7fca2eb7e3d5a3371312c2b6b56"}
Oct 30 19:30:13.211736 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:13.211707    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:13.213222 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:13.213199    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zhkqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:30:13.213341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:13.213244    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 19:30:14.212050 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:14.212022    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zhkqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:30:14.212417 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:14.212075    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 19:30:15.211906 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:15.211871    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zhkqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:30:15.212107 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:15.211929    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 19:30:16.212316 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:16.212280    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zhkqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:30:16.212780 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:16.212342    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 19:30:17.212395 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:17.212365    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zhkqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:30:17.212763 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:17.212411    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 19:30:18.212156 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:18.212118    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zhkqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 30 19:30:18.212327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:18.212198    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 30 19:30:19.883571 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:19.883541    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:19.884089 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:19.883939    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:19.884700 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:19.884684    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:30:19.889050 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:19.889036    4420 scope.go:122] "RemoveContainer" containerID="42b3f5668d4c3ce51e1aa120f0caf6d8a490a2e96bf4dea0e0d89278f10112ca"
Oct 30 19:30:20.099145 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:20.099103    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" podStartSLOduration=10.099090646 podStartE2EDuration="10.099090646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:12.668174207 +0000 UTC m=+859.879391352" watchObservedRunningTime="2026-10-30 19:30:20.099090646 +0000 UTC m=+867.310307790"
Oct 30 19:30:20.506324 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:20.506240    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb" event={"ID":"633766ff-cc60-41bc-b92b-c0ba2d8159a0","Type":"ContainerStarted","Data":"049d0caaf4a5229f6f172dabc9770f9b491835b8eb1b9ffcaa859a04ab33964f"}
Oct 30 19:30:21.512680 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:21.512651    4420 generic.go:356] "Generic (PLEG): container finished" podID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerID="049d0caaf4a5229f6f172dabc9770f9b491835b8eb1b9ffcaa859a04ab33964f" exitCode=60
Oct 30 19:30:21.513173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:21.512728    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb" event={"ID":"633766ff-cc60-41bc-b92b-c0ba2d8159a0","Type":"ContainerDied","Data":"049d0caaf4a5229f6f172dabc9770f9b491835b8eb1b9ffcaa859a04ab33964f"}
Oct 30 19:30:21.513173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:21.512802    4420 scope.go:122] "RemoveContainer" containerID="42b3f5668d4c3ce51e1aa120f0caf6d8a490a2e96bf4dea0e0d89278f10112ca"
Oct 30 19:30:21.513173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:21.513152    4420 scope.go:122] "RemoveContainer" containerID="049d0caaf4a5229f6f172dabc9770f9b491835b8eb1b9ffcaa859a04ab33964f"
Oct 30 19:30:21.513350 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:30:21.513333    4420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889810-7wzdb_odh-ai-gateway-infra(633766ff-cc60-41bc-b92b-c0ba2d8159a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0"
Oct 30 19:30:21.634045 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:21.634022    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"]
Oct 30 19:30:22.649130 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:22.649106    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"
Oct 30 19:30:22.682164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:22.682141    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633766ff-cc60-41bc-b92b-c0ba2d8159a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/633766ff-cc60-41bc-b92b-c0ba2d8159a0-maas-api-service-ca\") pod \"633766ff-cc60-41bc-b92b-c0ba2d8159a0\" (UID: \"633766ff-cc60-41bc-b92b-c0ba2d8159a0\") "
Oct 30 19:30:22.682287 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:22.682171    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633766ff-cc60-41bc-b92b-c0ba2d8159a0-kube-api-access-l624c\" (UniqueName: \"kubernetes.io/projected/633766ff-cc60-41bc-b92b-c0ba2d8159a0-kube-api-access-l624c\") pod \"633766ff-cc60-41bc-b92b-c0ba2d8159a0\" (UID: \"633766ff-cc60-41bc-b92b-c0ba2d8159a0\") "
Oct 30 19:30:22.682501 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:22.682480    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633766ff-cc60-41bc-b92b-c0ba2d8159a0-maas-api-service-ca" pod "633766ff-cc60-41bc-b92b-c0ba2d8159a0" (UID: "633766ff-cc60-41bc-b92b-c0ba2d8159a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:22.684344 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:22.684327    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633766ff-cc60-41bc-b92b-c0ba2d8159a0-kube-api-access-l624c" pod "633766ff-cc60-41bc-b92b-c0ba2d8159a0" (UID: "633766ff-cc60-41bc-b92b-c0ba2d8159a0"). InnerVolumeSpecName "kube-api-access-l624c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:22.783413 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:22.783348    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/633766ff-cc60-41bc-b92b-c0ba2d8159a0-maas-api-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:22.783413 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:22.783375    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l624c\" (UniqueName: \"kubernetes.io/projected/633766ff-cc60-41bc-b92b-c0ba2d8159a0-kube-api-access-l624c\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:30:23.523961 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:23.523929    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb" event={"ID":"633766ff-cc60-41bc-b92b-c0ba2d8159a0","Type":"ContainerDied","Data":"1daa25b90a7c7b75f9bd786e7ed05496a0e659a0edde05c0ab4215dd7c3d8316"}
Oct 30 19:30:23.524128 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:23.523982    4420 scope.go:122] "RemoveContainer" containerID="049d0caaf4a5229f6f172dabc9770f9b491835b8eb1b9ffcaa859a04ab33964f"
Oct 30 19:30:23.524128 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:23.523942    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"
Oct 30 19:30:23.686235 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:23.686209    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"]
Oct 30 19:30:23.688534 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:23.688514    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889810-7wzdb"]
Oct 30 19:30:24.894328 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:24.894295    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" path="/var/lib/kubelet/pods/633766ff-cc60-41bc-b92b-c0ba2d8159a0/volumes"
Oct 30 19:30:27.513563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.513530    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"]
Oct 30 19:30:27.513989 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.513976    4420 cpu_manager.go:414] "containerMap: removing container" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerName="cleanup"
Oct 30 19:30:27.514030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.513990    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerName="cleanup"
Oct 30 19:30:27.514030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.514002    4420 cpu_manager.go:414] "containerMap: removing container" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerName="cleanup"
Oct 30 19:30:27.514030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.514006    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerName="cleanup"
Oct 30 19:30:27.514030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.514013    4420 cpu_manager.go:414] "containerMap: removing container" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerName="cleanup"
Oct 30 19:30:27.514030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.514017    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerName="cleanup"
Oct 30 19:30:27.514192 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.514089    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerName="cleanup"
Oct 30 19:30:27.514192 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.514097    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerName="cleanup"
Oct 30 19:30:27.519482 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.519460    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:30:27.576947 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.576921    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"]
Oct 30 19:30:27.622918 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.622893    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-lw9gz\""
Oct 30 19:30:27.622918 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.622905    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 19:30:27.623127 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.622972    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 19:30:27.623514 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.623494    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e48596-4bb5-4d1b-881b-74503faccce2-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-2g49t\" (UID: \"33e48596-4bb5-4d1b-881b-74503faccce2\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:30:27.623619 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.623534    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44jlf\" (UniqueName: \"kubernetes.io/projected/33e48596-4bb5-4d1b-881b-74503faccce2-kube-api-access-44jlf\") pod \"ai-gateway-controller-5d9cd99f45-2g49t\" (UID: \"33e48596-4bb5-4d1b-881b-74503faccce2\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:30:27.725413 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.725373    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e48596-4bb5-4d1b-881b-74503faccce2-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-2g49t\" (UID: \"33e48596-4bb5-4d1b-881b-74503faccce2\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:30:27.725616 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.725498    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44jlf\" (UniqueName: \"kubernetes.io/projected/33e48596-4bb5-4d1b-881b-74503faccce2-kube-api-access-44jlf\") pod \"ai-gateway-controller-5d9cd99f45-2g49t\" (UID: \"33e48596-4bb5-4d1b-881b-74503faccce2\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:30:27.729860 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.729386    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e48596-4bb5-4d1b-881b-74503faccce2-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-2g49t\" (UID: \"33e48596-4bb5-4d1b-881b-74503faccce2\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:30:27.782138 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.782070    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44jlf\" (UniqueName: \"kubernetes.io/projected/33e48596-4bb5-4d1b-881b-74503faccce2-kube-api-access-44jlf\") pod \"ai-gateway-controller-5d9cd99f45-2g49t\" (UID: \"33e48596-4bb5-4d1b-881b-74503faccce2\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:30:27.833713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:27.833676    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:30:28.206168 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:28.206146    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"]
Oct 30 19:30:28.208026 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:30:28.207998    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e48596_4bb5_4d1b_881b_74503faccce2.slice/crio-18da8df27938686f67b56ba2bbe9179e725dbbf4049a39f106ee473d3caf5507 WatchSource:0}: Error finding container 18da8df27938686f67b56ba2bbe9179e725dbbf4049a39f106ee473d3caf5507: Status 404 returned error can't find the container with id 18da8df27938686f67b56ba2bbe9179e725dbbf4049a39f106ee473d3caf5507
Oct 30 19:30:28.549137 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:28.549065    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t" event={"ID":"33e48596-4bb5-4d1b-881b-74503faccce2","Type":"ContainerStarted","Data":"18da8df27938686f67b56ba2bbe9179e725dbbf4049a39f106ee473d3caf5507"}
Oct 30 19:30:33.574453 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:33.574421    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t" event={"ID":"33e48596-4bb5-4d1b-881b-74503faccce2","Type":"ContainerStarted","Data":"f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c"}
Oct 30 19:30:33.574919 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:33.574560    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:30:33.696052 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:33.695997    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t" podStartSLOduration=1.992732283 podStartE2EDuration="6.695979601s" totalImagesPullingTime="4.703247318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:33.694719536 +0000 UTC m=+880.905936682" watchObservedRunningTime="2026-10-30 19:30:33.695979601 +0000 UTC m=+880.907196746"
Oct 30 19:30:35.195465 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:30:35.195433    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:30:44.581566 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:44.581539    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:30:55.991172 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:55.991145    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"]
Oct 30 19:30:55.991608 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:55.991596    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="633766ff-cc60-41bc-b92b-c0ba2d8159a0" containerName="cleanup"
Oct 30 19:30:55.995754 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:55.995738    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.075374 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.075347    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.075514 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.075377    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.075514 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.075411    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.075637 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.075531    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6snq7\" (UniqueName: \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-kube-api-access-6snq7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.075637 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.075577    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc4d891-868b-425a-b272-f2a75d985419-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.075637 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.075618    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.075747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.075716    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.075787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.075751    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.075787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.075776    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc4d891-868b-425a-b272-f2a75d985419-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.096937 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.096912    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"]
Oct 30 19:30:56.177149 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177123    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177296 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177154    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6snq7\" (UniqueName: \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-kube-api-access-6snq7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177296 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177179    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc4d891-868b-425a-b272-f2a75d985419-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177296 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177238    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177460 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177315    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177460 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177349    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177460 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177386    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc4d891-868b-425a-b272-f2a75d985419-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177460 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177444    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177645 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177478    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177645 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177632    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177801 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177768    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177797    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.177931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177879    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.178015 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.177927    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc4d891-868b-425a-b272-f2a75d985419-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.179744 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.179719    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.180086 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.180065    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc4d891-868b-425a-b272-f2a75d985419-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.238738 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.238708    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.238861 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.238757    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6snq7\" (UniqueName: \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-kube-api-access-6snq7\") pod \"maas-default-gateway-openshift-default-6db586dd6c-p9fnh\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.308499 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.308441    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:56.731950 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.731921    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"]
Oct 30 19:30:56.739646 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:30:56.739619    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc4d891_868b_425a_b272_f2a75d985419.slice/crio-853e9d16d30cf8856fd09c42eb0e9dcc13fb0ae19ab031884f87294c6101e3dd WatchSource:0}: Error finding container 853e9d16d30cf8856fd09c42eb0e9dcc13fb0ae19ab031884f87294c6101e3dd: Status 404 returned error can't find the container with id 853e9d16d30cf8856fd09c42eb0e9dcc13fb0ae19ab031884f87294c6101e3dd
Oct 30 19:30:56.741807 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.741775    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:56.741905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.741861    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:56.741942 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:56.741908    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:30:57.698749 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:57.698718    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" event={"ID":"1cc4d891-868b-425a-b272-f2a75d985419","Type":"ContainerStarted","Data":"b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc"}
Oct 30 19:30:57.699189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:57.698766    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" event={"ID":"1cc4d891-868b-425a-b272-f2a75d985419","Type":"ContainerStarted","Data":"853e9d16d30cf8856fd09c42eb0e9dcc13fb0ae19ab031884f87294c6101e3dd"}
Oct 30 19:30:57.819592 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:57.819464    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" podStartSLOduration=2.819448957 podStartE2EDuration="2.819448957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:57.817981992 +0000 UTC m=+905.029199140" watchObservedRunningTime="2026-10-30 19:30:57.819448957 +0000 UTC m=+905.030666102"
Oct 30 19:30:58.309691 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:58.309661    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:30:58.311063 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:58.311038    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p9fnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:30:58.311176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:58.311084    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" podUID="1cc4d891-868b-425a-b272-f2a75d985419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:30:59.309716 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:59.309672    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p9fnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:30:59.310245 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:30:59.309740    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" podUID="1cc4d891-868b-425a-b272-f2a75d985419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:31:00.017309 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.017282    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"]
Oct 30 19:31:00.017476 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.017461    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t" podUID="33e48596-4bb5-4d1b-881b-74503faccce2" containerName="manager" containerID="cri-o://f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c" gracePeriod=30
Oct 30 19:31:00.308436 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.308416    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:31:00.308843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.308798    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p9fnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:31:00.308933 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.308870    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" podUID="1cc4d891-868b-425a-b272-f2a75d985419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:31:00.420640 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.420613    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e48596-4bb5-4d1b-881b-74503faccce2-kube-api-access-44jlf\" (UniqueName: \"kubernetes.io/projected/33e48596-4bb5-4d1b-881b-74503faccce2-kube-api-access-44jlf\") pod \"33e48596-4bb5-4d1b-881b-74503faccce2\" (UID: \"33e48596-4bb5-4d1b-881b-74503faccce2\") "
Oct 30 19:31:00.421086 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.420681    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e48596-4bb5-4d1b-881b-74503faccce2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e48596-4bb5-4d1b-881b-74503faccce2-metrics-certs\") pod \"33e48596-4bb5-4d1b-881b-74503faccce2\" (UID: \"33e48596-4bb5-4d1b-881b-74503faccce2\") "
Oct 30 19:31:00.422752 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.422722    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e48596-4bb5-4d1b-881b-74503faccce2-kube-api-access-44jlf" pod "33e48596-4bb5-4d1b-881b-74503faccce2" (UID: "33e48596-4bb5-4d1b-881b-74503faccce2"). InnerVolumeSpecName "kube-api-access-44jlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:31:00.422879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.422770    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e48596-4bb5-4d1b-881b-74503faccce2-metrics-certs" pod "33e48596-4bb5-4d1b-881b-74503faccce2" (UID: "33e48596-4bb5-4d1b-881b-74503faccce2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:31:00.521750 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.521720    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44jlf\" (UniqueName: \"kubernetes.io/projected/33e48596-4bb5-4d1b-881b-74503faccce2-kube-api-access-44jlf\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:00.521750 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.521746    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33e48596-4bb5-4d1b-881b-74503faccce2-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:31:00.714203 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.714163    4420 generic.go:356] "Generic (PLEG): container finished" podID="33e48596-4bb5-4d1b-881b-74503faccce2" containerID="f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c" exitCode=0
Oct 30 19:31:00.714399 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.714226    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"
Oct 30 19:31:00.714399 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.714246    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t" event={"ID":"33e48596-4bb5-4d1b-881b-74503faccce2","Type":"ContainerDied","Data":"f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c"}
Oct 30 19:31:00.714399 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.714313    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-2g49t" event={"ID":"33e48596-4bb5-4d1b-881b-74503faccce2","Type":"ContainerDied","Data":"18da8df27938686f67b56ba2bbe9179e725dbbf4049a39f106ee473d3caf5507"}
Oct 30 19:31:00.714399 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.714342    4420 scope.go:122] "RemoveContainer" containerID="f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c"
Oct 30 19:31:00.723966 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.723947    4420 scope.go:122] "RemoveContainer" containerID="f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c"
Oct 30 19:31:00.724227 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:31:00.724207    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c\": container with ID starting with f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c not found: ID does not exist" containerID="f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c"
Oct 30 19:31:00.724301 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.724240    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c"} err="failed to get container status \"f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c\": rpc error: code = NotFound desc = could not find container \"f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c\": container with ID starting with f7f865e3a4d250db383d53ca64b774d7c1675c9cf911ec88617a199552fb903c not found: ID does not exist"
Oct 30 19:31:00.883324 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.882261    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"]
Oct 30 19:31:00.930299 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:00.930273    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-2g49t"]
Oct 30 19:31:01.309091 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:01.309062    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p9fnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:31:01.309258 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:01.309108    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" podUID="1cc4d891-868b-425a-b272-f2a75d985419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:31:02.309081 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:02.309050    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p9fnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:31:02.309456 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:02.309140    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" podUID="1cc4d891-868b-425a-b272-f2a75d985419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:31:02.897328 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:02.897293    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e48596-4bb5-4d1b-881b-74503faccce2" path="/var/lib/kubelet/pods/33e48596-4bb5-4d1b-881b-74503faccce2/volumes"
Oct 30 19:31:03.309510 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:03.309424    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p9fnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:31:03.309510 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:03.309467    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" podUID="1cc4d891-868b-425a-b272-f2a75d985419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:31:04.308952 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:04.308916    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-p9fnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 19:31:04.309250 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:04.308961    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" podUID="1cc4d891-868b-425a-b272-f2a75d985419" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 19:31:05.196756 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:31:05.196717    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:31:05.319758 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:05.319728    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:31:05.320274 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:05.320221    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:31:05.321217 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:31:05.321194    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:31:35.195574 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:31:35.195492    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:32:01.790723 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.790692    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"]
Oct 30 19:32:01.791164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.791150    4420 cpu_manager.go:414] "containerMap: removing container" podUID="33e48596-4bb5-4d1b-881b-74503faccce2" containerName="manager"
Oct 30 19:32:01.791211 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.791164    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e48596-4bb5-4d1b-881b-74503faccce2" containerName="manager"
Oct 30 19:32:01.791245 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.791236    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e48596-4bb5-4d1b-881b-74503faccce2" containerName="manager"
Oct 30 19:32:01.795800 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.795785    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.845740 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.845704    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phg68\" (UniqueName: \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-kube-api-access-phg68\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.845898 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.845763    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.845898 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.845865    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.845976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.845947    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f4fdbb-d865-42f6-92cd-331a5468deed-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.846010 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.845984    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.846044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.846007    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.846138 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.846115    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.846180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.846160    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.846221 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.846198    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.869502 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.869472    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"]
Oct 30 19:32:01.896529 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.896507    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ada8d7-openshift-default-dockercfg-wtmwk\""
Oct 30 19:32:01.947065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.947030    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.947065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.947080    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.949334 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.947156    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f4fdbb-d865-42f6-92cd-331a5468deed-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.949334 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.947213    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.949334 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.947242    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.949334 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.947271    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.949334 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.947305    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.949334 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.947337    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.949334 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.947378    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phg68\" (UniqueName: \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-kube-api-access-phg68\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.949334 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.948377    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f4fdbb-d865-42f6-92cd-331a5468deed-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.949978 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.949936    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.950205 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.949985    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.950562 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.950535    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.950787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.950768    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.952670 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.951974    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:01.960357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:01.960332    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:02.005338 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.005309    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phg68\" (UniqueName: \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-kube-api-access-phg68\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:02.006597 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.006576    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:02.108365 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.108300    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:02.109596 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.109574    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"]
Oct 30 19:32:02.113158 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.113137    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.148473 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.148444    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.148630 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.148481    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b0996b-c55b-4e37-8220-1302701f6260-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.148630 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.148605    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.148630 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.148626    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b0996b-c55b-4e37-8220-1302701f6260-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.148796 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.148648    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.148796 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.148679    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.148796 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.148693    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.148796 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.148714    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgb6j\" (UniqueName: \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-kube-api-access-mgb6j\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.148796 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.148767    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.166502 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.166395    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d52df0-openshift-default-dockercfg-dwj6l\""
Oct 30 19:32:02.180610 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.179135    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"]
Oct 30 19:32:02.249282 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.249252    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.249459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.249306    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.249459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.249332    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b0996b-c55b-4e37-8220-1302701f6260-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.249459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.249400    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.249459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.249428    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b0996b-c55b-4e37-8220-1302701f6260-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.249459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.249453    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.249763 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.249688    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.249923 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.249898    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.250044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.250013    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.250163 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.250052    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.250163 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.250079    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgb6j\" (UniqueName: \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-kube-api-access-mgb6j\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.250163 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.250103    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b0996b-c55b-4e37-8220-1302701f6260-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.250333 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.250284    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.250649 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.250626    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.252159 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.252123    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b0996b-c55b-4e37-8220-1302701f6260-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.252285 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.252270    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.321045 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.321019    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.321220 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.321203    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgb6j\" (UniqueName: \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-kube-api-access-mgb6j\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.438440 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.438406    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:02.562125 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.562102    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"]
Oct 30 19:32:02.564381 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:32:02.564354    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f4fdbb_d865_42f6_92cd_331a5468deed.slice/crio-af7b073d8ece474ee55166bae88cfab0b2bc8ffd530617c85036c12d57c4f1a3 WatchSource:0}: Error finding container af7b073d8ece474ee55166bae88cfab0b2bc8ffd530617c85036c12d57c4f1a3: Status 404 returned error can't find the container with id af7b073d8ece474ee55166bae88cfab0b2bc8ffd530617c85036c12d57c4f1a3
Oct 30 19:32:02.567127 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.566901    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:02.567127 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.566989    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:02.567127 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.567030    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:02.815968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.815941    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"]
Oct 30 19:32:02.820050 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:32:02.820011    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b0996b_c55b_4e37_8220_1302701f6260.slice/crio-e21fefaca12136b95399b84c87fa15f1832b0b5f3181fc43965aab88c6d0e85c WatchSource:0}: Error finding container e21fefaca12136b95399b84c87fa15f1832b0b5f3181fc43965aab88c6d0e85c: Status 404 returned error can't find the container with id e21fefaca12136b95399b84c87fa15f1832b0b5f3181fc43965aab88c6d0e85c
Oct 30 19:32:02.822107 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.822075    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:02.822226 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.822159    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:02.822226 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:02.822212    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:03.002074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:03.001800    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" event={"ID":"29b0996b-c55b-4e37-8220-1302701f6260","Type":"ContainerStarted","Data":"69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945"}
Oct 30 19:32:03.002074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:03.001890    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" event={"ID":"29b0996b-c55b-4e37-8220-1302701f6260","Type":"ContainerStarted","Data":"e21fefaca12136b95399b84c87fa15f1832b0b5f3181fc43965aab88c6d0e85c"}
Oct 30 19:32:03.003650 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:03.003621    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" event={"ID":"a7f4fdbb-d865-42f6-92cd-331a5468deed","Type":"ContainerStarted","Data":"60e21861f120f21af8df152a3500cffd24814e8182ebadcc1056a106d838e143"}
Oct 30 19:32:03.003757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:03.003675    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" event={"ID":"a7f4fdbb-d865-42f6-92cd-331a5468deed","Type":"ContainerStarted","Data":"af7b073d8ece474ee55166bae88cfab0b2bc8ffd530617c85036c12d57c4f1a3"}
Oct 30 19:32:03.108697 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:03.108624    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:03.113090 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:03.113072    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:03.122003 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:03.121964    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" podStartSLOduration=1.121951603 podStartE2EDuration="1.121951603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:03.120604215 +0000 UTC m=+970.331821364" watchObservedRunningTime="2026-10-30 19:32:03.121951603 +0000 UTC m=+970.333168747"
Oct 30 19:32:03.240434 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:03.240374    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" podStartSLOduration=2.240354923 podStartE2EDuration="2.240354923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:03.23834294 +0000 UTC m=+970.449560103" watchObservedRunningTime="2026-10-30 19:32:03.240354923 +0000 UTC m=+970.451572067"
Oct 30 19:32:03.439498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:03.439469    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:03.444543 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:03.444522    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:04.009210 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:04.009177    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:04.009603 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:04.009238    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:04.009828 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:04.009792    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:04.009967 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:04.009918    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:05.195668 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:32:05.195636    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:32:11.455001 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.454972    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"]
Oct 30 19:32:11.458928 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.458907    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.514216 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.514188    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a3f9b7-openshift-default-dockercfg-9v6f2\""
Oct 30 19:32:11.531806 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.531774    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.531806 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.531836    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5h8d\" (UniqueName: \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-kube-api-access-x5h8d\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.532069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.531919    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.532069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.531966    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.532069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.532038    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.532293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.532089    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.532293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.532118    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d86524-a126-407d-85ed-5a62b852046f-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.532293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.532207    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.532293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.532238    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d86524-a126-407d-85ed-5a62b852046f-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.566632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.566602    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"]
Oct 30 19:32:11.633787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.633757    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.633985 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.633793    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5h8d\" (UniqueName: \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-kube-api-access-x5h8d\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.633985 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.633861    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.633985 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.633899    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.633985 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.633968    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.634212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.634010    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.634212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.634047    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d86524-a126-407d-85ed-5a62b852046f-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.634212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.634140    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.634212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.634204    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d86524-a126-407d-85ed-5a62b852046f-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.634406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.634225    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.634406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.634251    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.634406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.634353    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.634570 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.634507    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.634954 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.634931    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d86524-a126-407d-85ed-5a62b852046f-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.636619 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.636598    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d86524-a126-407d-85ed-5a62b852046f-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.636735 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.636643    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.690303 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.690282    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.690704 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.690677    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5h8d\" (UniqueName: \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-kube-api-access-x5h8d\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.769169 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.769098    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"]
Oct 30 19:32:11.772096 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.772062    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:11.773988 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.773968    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.847897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.847851    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-2d9861-openshift-default-dockercfg-ngs2l\""
Oct 30 19:32:11.848495 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.848471    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.848641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.848576    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.848703 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.848654    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.848703 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.848683    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.848807 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.848715    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a35842-06c7-43f1-b1b5-cdabd86db545-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.848807 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.848752    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.849891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.848805    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjr7\" (UniqueName: \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-kube-api-access-jrjr7\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.850029 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.849934    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.850029 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.849962    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.851381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.851357    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"]
Oct 30 19:32:11.951113 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.951068    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.951298 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.951188    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.951298 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.951242    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.951298 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.951281    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.951482 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.951327    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a35842-06c7-43f1-b1b5-cdabd86db545-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.951482 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.951389    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.951482 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.951466    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjr7\" (UniqueName: \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-kube-api-access-jrjr7\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.951645 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.951520    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.951645 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.951552    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.951976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.951956    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a35842-06c7-43f1-b1b5-cdabd86db545-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.952310 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.952287    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.952310 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.952294    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.952310 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.952307    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.952490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.952393    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.953602 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.953585    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:11.953829 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:11.953801    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:12.022155 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.022097    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:12.023460 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.023443    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjr7\" (UniqueName: \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-kube-api-access-jrjr7\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:12.098921 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.098885    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:12.226830 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.226787    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"]
Oct 30 19:32:12.230528 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.230512    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.238852 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.238807    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"]
Oct 30 19:32:12.240554 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:32:12.240532    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d86524_a126_407d_85ed_5a62b852046f.slice/crio-e91d3336a42831c73100ad55251fe2470b539b49b8dec42b62a71b6e684180fb WatchSource:0}: Error finding container e91d3336a42831c73100ad55251fe2470b539b49b8dec42b62a71b6e684180fb: Status 404 returned error can't find the container with id e91d3336a42831c73100ad55251fe2470b539b49b8dec42b62a71b6e684180fb
Oct 30 19:32:12.242777 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.242751    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:12.242882 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.242805    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:12.242882 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.242855    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:12.286109 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.286089    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-ada8d7-serving-cert\""
Oct 30 19:32:12.286319 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.286299    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 19:32:12.303360 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.303337    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"]
Oct 30 19:32:12.355828 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.355783    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a210fc35-eee1-44f1-9dc3-eb66dc04f288-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.355998 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.355915    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.356073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.356025    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-metrics-certs\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.356152 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.356137    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2wjp\" (UniqueName: \"kubernetes.io/projected/a210fc35-eee1-44f1-9dc3-eb66dc04f288-kube-api-access-n2wjp\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.386907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.386878    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8c9cv\""
Oct 30 19:32:12.457479 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.457446    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a210fc35-eee1-44f1-9dc3-eb66dc04f288-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.457878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.457509    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.457878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.457556    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-metrics-certs\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.457878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.457616    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2wjp\" (UniqueName: \"kubernetes.io/projected/a210fc35-eee1-44f1-9dc3-eb66dc04f288-kube-api-access-n2wjp\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.457878 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:32:12.457678    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-serving-cert: secret "maas-api-e2e-worker-w3-ada8d7-serving-cert" not found
Oct 30 19:32:12.457878 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:32:12.457754    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-maas-api-tls podName:a210fc35-eee1-44f1-9dc3-eb66dc04f288 nodeName:}" failed. No retries permitted until 2026-10-30 19:32:12.957733232 +0000 UTC m=+980.168950368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-maas-api-tls") pod "maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9" (UID: "a210fc35-eee1-44f1-9dc3-eb66dc04f288") : secret "maas-api-e2e-worker-w3-ada8d7-serving-cert" not found
Oct 30 19:32:12.458224 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.458206    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a210fc35-eee1-44f1-9dc3-eb66dc04f288-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.460112 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.460094    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-metrics-certs\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.481304 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.481259    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"]
Oct 30 19:32:12.483533 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:32:12.483508    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a35842_06c7_43f1_b1b5_cdabd86db545.slice/crio-b5eba0aa24a2cddda41c447af41965dd7a03d02020e96b780ccb960ea513ce5f WatchSource:0}: Error finding container b5eba0aa24a2cddda41c447af41965dd7a03d02020e96b780ccb960ea513ce5f: Status 404 returned error can't find the container with id b5eba0aa24a2cddda41c447af41965dd7a03d02020e96b780ccb960ea513ce5f
Oct 30 19:32:12.485876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.485842    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:12.485980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.485919    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:12.485980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.485965    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:12.516547 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.516526    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2wjp\" (UniqueName: \"kubernetes.io/projected/a210fc35-eee1-44f1-9dc3-eb66dc04f288-kube-api-access-n2wjp\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.963023 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.962991    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:12.965394 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:12.965376    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:13.050409 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.050376    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" event={"ID":"36d86524-a126-407d-85ed-5a62b852046f","Type":"ContainerStarted","Data":"59ece95686bb3dc10cbdd236182b2afdb39a93b02c77570e42cf4c8d4db7711f"}
Oct 30 19:32:13.050538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.050424    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" event={"ID":"36d86524-a126-407d-85ed-5a62b852046f","Type":"ContainerStarted","Data":"e91d3336a42831c73100ad55251fe2470b539b49b8dec42b62a71b6e684180fb"}
Oct 30 19:32:13.052135 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.052110    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" event={"ID":"a5a35842-06c7-43f1-b1b5-cdabd86db545","Type":"ContainerStarted","Data":"161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e"}
Oct 30 19:32:13.052247 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.052149    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" event={"ID":"a5a35842-06c7-43f1-b1b5-cdabd86db545","Type":"ContainerStarted","Data":"b5eba0aa24a2cddda41c447af41965dd7a03d02020e96b780ccb960ea513ce5f"}
Oct 30 19:32:13.099529 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.099509    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:13.104994 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.104976    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:13.144994 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.144972    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:13.170069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.170021    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" podStartSLOduration=2.170002243 podStartE2EDuration="2.170002243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:13.166009627 +0000 UTC m=+980.377226800" watchObservedRunningTime="2026-10-30 19:32:13.170002243 +0000 UTC m=+980.381219388"
Oct 30 19:32:13.287998 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.287277    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" podStartSLOduration=2.28726444 podStartE2EDuration="2.28726444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:13.282255884 +0000 UTC m=+980.493473030" watchObservedRunningTime="2026-10-30 19:32:13.28726444 +0000 UTC m=+980.498481585"
Oct 30 19:32:13.518108 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.518084    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"]
Oct 30 19:32:13.520356 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:32:13.520328    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda210fc35_eee1_44f1_9dc3_eb66dc04f288.slice/crio-f353e601c66de306066a8c97597cd7dd533034df2c8fba6211f47a9797d5dbee WatchSource:0}: Error finding container f353e601c66de306066a8c97597cd7dd533034df2c8fba6211f47a9797d5dbee: Status 404 returned error can't find the container with id f353e601c66de306066a8c97597cd7dd533034df2c8fba6211f47a9797d5dbee
Oct 30 19:32:13.521925 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.521906    4420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:32:13.772277 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.772246    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:13.776901 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:13.776877    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:14.058013 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:14.057940    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9" event={"ID":"a210fc35-eee1-44f1-9dc3-eb66dc04f288","Type":"ContainerStarted","Data":"f353e601c66de306066a8c97597cd7dd533034df2c8fba6211f47a9797d5dbee"}
Oct 30 19:32:14.058363 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:14.058337    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:14.058527 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:14.058508    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:14.059238 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:14.059221    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:32:14.059331 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:14.059268    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:32:16.069912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:16.069878    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9" event={"ID":"a210fc35-eee1-44f1-9dc3-eb66dc04f288","Type":"ContainerStarted","Data":"b4d215fbc22e1cb8094ec94f2c62bfa9f2573561ba3f5355a9b5d50535ffc073"}
Oct 30 19:32:16.070393 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:16.070100    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:16.184779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:16.184733    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9" podStartSLOduration=2.19895286 podStartE2EDuration="4.184719847s" totalImagesPullingTime="1.985766987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:16.182990184 +0000 UTC m=+983.394207339" watchObservedRunningTime="2026-10-30 19:32:16.184719847 +0000 UTC m=+983.395936991"
Oct 30 19:32:21.781008 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.780966    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"]
Oct 30 19:32:21.785027 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.785008    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.863488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.863451    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"]
Oct 30 19:32:21.863893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.863870    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.863939 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.863902    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.864015 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.863995    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.864084 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.864066    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.864120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.864108    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a417e0-9ddc-4187-aaf8-632e3e20401f-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.864230 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.864217    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.864271 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.864255    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdz9f\" (UniqueName: \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-kube-api-access-wdz9f\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.864317 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.864279    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.864427 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.864406    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.885913 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.885866    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b5178f-openshift-default-dockercfg-k2spr\""
Oct 30 19:32:21.965035 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965009    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.965215 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965119    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.965215 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965177    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.965447 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965262    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.965447 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965287    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.965447 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965328    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a417e0-9ddc-4187-aaf8-632e3e20401f-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.965447 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965411    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.965724 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965461    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdz9f\" (UniqueName: \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-kube-api-access-wdz9f\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.965724 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965499    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.965724 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965665    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.965933 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965804    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.965933 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.965901    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.966355 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.966331    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a417e0-9ddc-4187-aaf8-632e3e20401f-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.966493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.966477    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.967998 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.967977    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:21.968412 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:21.968395    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:22.025686 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:22.025078    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:22.025686 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:22.025214    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdz9f\" (UniqueName: \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-kube-api-access-wdz9f\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:22.080967 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:22.080899    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 19:32:22.097713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:22.097687    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:22.523970 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:22.523940    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"]
Oct 30 19:32:22.527376 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:32:22.527346    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a417e0_9ddc_4187_aaf8_632e3e20401f.slice/crio-de97e924432d865a1889a7fb141df7e97c36439c80f47dc9cf42f68a15813056 WatchSource:0}: Error finding container de97e924432d865a1889a7fb141df7e97c36439c80f47dc9cf42f68a15813056: Status 404 returned error can't find the container with id de97e924432d865a1889a7fb141df7e97c36439c80f47dc9cf42f68a15813056
Oct 30 19:32:22.529622 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:22.529590    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:22.529683 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:22.529659    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:22.529715 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:22.529693    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:23.103886 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:23.103852    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" event={"ID":"90a417e0-9ddc-4187-aaf8-632e3e20401f","Type":"ContainerStarted","Data":"f5ea3b5ae48e1b6d63ab68f2a812ff11c23b88d287e4ee723bc0be585595e260"}
Oct 30 19:32:23.104277 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:23.103909    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" event={"ID":"90a417e0-9ddc-4187-aaf8-632e3e20401f","Type":"ContainerStarted","Data":"de97e924432d865a1889a7fb141df7e97c36439c80f47dc9cf42f68a15813056"}
Oct 30 19:32:23.221571 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:23.221523    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" podStartSLOduration=2.2215100740000002 podStartE2EDuration="2.221510074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:23.219063071 +0000 UTC m=+990.430280217" watchObservedRunningTime="2026-10-30 19:32:23.221510074 +0000 UTC m=+990.432727218"
Oct 30 19:32:24.098178 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:24.098143    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:24.102653 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:24.102628    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:24.109042 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:24.109013    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:24.109667 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:24.109653    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:32:31.739486 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:31.739448    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"]
Oct 30 19:32:31.744093 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:31.744071    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 19:32:31.862415 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:31.860886    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-a3f9b7\""
Oct 30 19:32:31.862415 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:31.861060    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-a3f9b7-dockercfg-cs4qp\""
Oct 30 19:32:31.865545 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:31.865506    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"]
Oct 30 19:32:31.892356 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:31.892327    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fksdf\" (UniqueName: \"kubernetes.io/projected/c21754ed-109f-45af-ad88-de64f423c0fc-kube-api-access-fksdf\") pod \"payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh\" (UID: \"c21754ed-109f-45af-ad88-de64f423c0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 19:32:31.892516 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:31.892397    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21754ed-109f-45af-ad88-de64f423c0fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh\" (UID: \"c21754ed-109f-45af-ad88-de64f423c0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 19:32:31.993184 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:31.993105    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21754ed-109f-45af-ad88-de64f423c0fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh\" (UID: \"c21754ed-109f-45af-ad88-de64f423c0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 19:32:31.993315 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:31.993250    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fksdf\" (UniqueName: \"kubernetes.io/projected/c21754ed-109f-45af-ad88-de64f423c0fc-kube-api-access-fksdf\") pod \"payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh\" (UID: \"c21754ed-109f-45af-ad88-de64f423c0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 19:32:31.993977 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:31.993957    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21754ed-109f-45af-ad88-de64f423c0fc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh\" (UID: \"c21754ed-109f-45af-ad88-de64f423c0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 19:32:32.049577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:32.049551    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fksdf\" (UniqueName: \"kubernetes.io/projected/c21754ed-109f-45af-ad88-de64f423c0fc-kube-api-access-fksdf\") pod \"payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh\" (UID: \"c21754ed-109f-45af-ad88-de64f423c0fc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 19:32:32.058305 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:32.058284    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 19:32:32.434431 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:32.434409    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"]
Oct 30 19:32:32.436660 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:32:32.436621    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21754ed_109f_45af_ad88_de64f423c0fc.slice/crio-8d910200047c204db1edef2f69f789d63f4d74b2e6a5477b6202582ec5a12bde WatchSource:0}: Error finding container 8d910200047c204db1edef2f69f789d63f4d74b2e6a5477b6202582ec5a12bde: Status 404 returned error can't find the container with id 8d910200047c204db1edef2f69f789d63f4d74b2e6a5477b6202582ec5a12bde
Oct 30 19:32:33.158547 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:33.158499    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh" event={"ID":"c21754ed-109f-45af-ad88-de64f423c0fc","Type":"ContainerStarted","Data":"8d910200047c204db1edef2f69f789d63f4d74b2e6a5477b6202582ec5a12bde"}
Oct 30 19:32:35.051009 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:35.050966    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:32:35.051431 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:35.051045    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:32:35.220562 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:32:35.220516    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:32:36.207621 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:36.207588    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh" event={"ID":"c21754ed-109f-45af-ad88-de64f423c0fc","Type":"ContainerStarted","Data":"5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9"}
Oct 30 19:32:36.208146 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:36.207687    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 19:32:36.209235 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:36.209215    4420 generic.go:356] "Generic (PLEG): container finished" podID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerID="60e21861f120f21af8df152a3500cffd24814e8182ebadcc1056a106d838e143" exitCode=0
Oct 30 19:32:36.209337 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:36.209287    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" event={"ID":"a7f4fdbb-d865-42f6-92cd-331a5468deed","Type":"ContainerDied","Data":"60e21861f120f21af8df152a3500cffd24814e8182ebadcc1056a106d838e143"}
Oct 30 19:32:36.209624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:36.209611    4420 scope.go:122] "RemoveContainer" containerID="60e21861f120f21af8df152a3500cffd24814e8182ebadcc1056a106d838e143"
Oct 30 19:32:36.210920 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:36.210899    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:36.210974 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:36.210957    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:36.211007 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:36.210988    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:36.324482 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:36.323789    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh" podStartSLOduration=2.063620683 podStartE2EDuration="5.323772834s" totalImagesPullingTime="3.260152151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:36.322637283 +0000 UTC m=+1003.533854429" watchObservedRunningTime="2026-10-30 19:32:36.323772834 +0000 UTC m=+1003.534989979"
Oct 30 19:32:37.217355 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:37.217325    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" event={"ID":"a7f4fdbb-d865-42f6-92cd-331a5468deed","Type":"ContainerStarted","Data":"fc073dc3f7add73f1dca686f0313b54211ff32fb898df30548b743ddd318e4fe"}
Oct 30 19:32:38.109441 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.109398    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:32:38.109621 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.109460    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:32:38.109746 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.109725    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:38.246339 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.246311    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"]
Oct 30 19:32:38.252676 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.252657    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:38.315302 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.315273    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-a3f9b7-serving-cert\""
Oct 30 19:32:38.317985 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.317896    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"]
Oct 30 19:32:38.406840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.403548    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptm5q\" (UniqueName: \"kubernetes.io/projected/2b8980fb-7181-4478-a3f1-89d285253ff7-kube-api-access-ptm5q\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:38.406840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.403648    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8980fb-7181-4478-a3f1-89d285253ff7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:38.406840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.403737    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-metrics-certs\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:38.406840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.403776    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:38.508600 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.508557    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptm5q\" (UniqueName: \"kubernetes.io/projected/2b8980fb-7181-4478-a3f1-89d285253ff7-kube-api-access-ptm5q\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:38.508832 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.508642    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8980fb-7181-4478-a3f1-89d285253ff7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:38.508832 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.508711    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-metrics-certs\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:38.508832 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.508749    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:38.509034 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:32:38.508928    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-serving-cert: secret "maas-api-e2e-worker-w2-a3f9b7-serving-cert" not found
Oct 30 19:32:38.509034 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:32:38.509001    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-maas-api-tls podName:2b8980fb-7181-4478-a3f1-89d285253ff7 nodeName:}" failed. No retries permitted until 2026-10-30 19:32:39.008982899 +0000 UTC m=+1006.220200037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-maas-api-tls") pod "maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94" (UID: "2b8980fb-7181-4478-a3f1-89d285253ff7") : secret "maas-api-e2e-worker-w2-a3f9b7-serving-cert" not found
Oct 30 19:32:38.509567 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.509542    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8980fb-7181-4478-a3f1-89d285253ff7-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:38.513621 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.513596    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-metrics-certs\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:38.608883 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:38.604312    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptm5q\" (UniqueName: \"kubernetes.io/projected/2b8980fb-7181-4478-a3f1-89d285253ff7-kube-api-access-ptm5q\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:39.036067 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:39.036033    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:39.038953 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:39.038925    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-maas-api-tls\") pod \"maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:39.110019 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:39.109885    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:32:39.110019 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:39.109980    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:32:39.208863 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:39.203278    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:39.646057 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:39.646028    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"]
Oct 30 19:32:39.650676 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:32:39.650647    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8980fb_7181_4478_a3f1_89d285253ff7.slice/crio-759e2afb04dae7dd7ada3b98c8240231b35380c64d46d26deb40bed1f2ccdd66 WatchSource:0}: Error finding container 759e2afb04dae7dd7ada3b98c8240231b35380c64d46d26deb40bed1f2ccdd66: Status 404 returned error can't find the container with id 759e2afb04dae7dd7ada3b98c8240231b35380c64d46d26deb40bed1f2ccdd66
Oct 30 19:32:40.110063 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:40.109988    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:32:40.110063 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:40.110039    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:32:40.237095 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:40.237067    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94" event={"ID":"2b8980fb-7181-4478-a3f1-89d285253ff7","Type":"ContainerStarted","Data":"759e2afb04dae7dd7ada3b98c8240231b35380c64d46d26deb40bed1f2ccdd66"}
Oct 30 19:32:41.109528 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:41.109485    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:32:41.110029 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:41.109544    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:32:41.244391 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:41.244346    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94" event={"ID":"2b8980fb-7181-4478-a3f1-89d285253ff7","Type":"ContainerStarted","Data":"be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1"}
Oct 30 19:32:41.244562 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:41.244422    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:42.109228 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:42.109191    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 19:32:42.109427 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:42.109243    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 19:32:43.113204 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.113180    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:43.113754 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.113471    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:43.114152 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.114133    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:32:43.283061 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.282999    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94" podStartSLOduration=4.342298769 podStartE2EDuration="5.282979861s" totalImagesPullingTime="940.681092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:41.366240594 +0000 UTC m=+1008.577457739" watchObservedRunningTime="2026-10-30 19:32:43.282979861 +0000 UTC m=+1010.494197050"
Oct 30 19:32:43.710098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.710038    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"]
Oct 30 19:32:43.731921 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.731897    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:43.771185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.771126    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"]
Oct 30 19:32:43.783239 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.783218    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-2d9861-serving-cert\""
Oct 30 19:32:43.816702 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.816674    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:43.816806 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.816717    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbdd\" (UniqueName: \"kubernetes.io/projected/82260772-1e21-46a5-bc36-4c4cc7ec8f75-kube-api-access-gwbdd\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:43.816806 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.816741    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-metrics-certs\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:43.816968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.816953    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82260772-1e21-46a5-bc36-4c4cc7ec8f75-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:43.918341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.918307    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:43.918504 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.918365    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbdd\" (UniqueName: \"kubernetes.io/projected/82260772-1e21-46a5-bc36-4c4cc7ec8f75-kube-api-access-gwbdd\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:43.918504 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.918394    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-metrics-certs\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:43.918504 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:32:43.918452    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-serving-cert: secret "maas-api-e2e-worker-w0-2d9861-serving-cert" not found
Oct 30 19:32:43.918669 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:32:43.918523    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-maas-api-tls podName:82260772-1e21-46a5-bc36-4c4cc7ec8f75 nodeName:}" failed. No retries permitted until 2026-10-30 19:32:44.418503407 +0000 UTC m=+1011.629720533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-maas-api-tls") pod "maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f" (UID: "82260772-1e21-46a5-bc36-4c4cc7ec8f75") : secret "maas-api-e2e-worker-w0-2d9861-serving-cert" not found
Oct 30 19:32:43.918669 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.918551    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82260772-1e21-46a5-bc36-4c4cc7ec8f75-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:43.919204 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.919185    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82260772-1e21-46a5-bc36-4c4cc7ec8f75-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:43.920884 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.920864    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-metrics-certs\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:43.975483 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:43.975451    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbdd\" (UniqueName: \"kubernetes.io/projected/82260772-1e21-46a5-bc36-4c4cc7ec8f75-kube-api-access-gwbdd\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:44.277367 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:44.277336    4420 generic.go:356] "Generic (PLEG): container finished" podID="29b0996b-c55b-4e37-8220-1302701f6260" containerID="69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945" exitCode=0
Oct 30 19:32:44.277882 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:44.277411    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" event={"ID":"29b0996b-c55b-4e37-8220-1302701f6260","Type":"ContainerDied","Data":"69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945"}
Oct 30 19:32:44.277882 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:44.277844    4420 scope.go:122] "RemoveContainer" containerID="69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945"
Oct 30 19:32:44.279101 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:44.279075    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:44.279166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:44.279145    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:44.279200 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:44.279181    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:32:44.423675 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:44.423643    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:44.427213 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:44.427190    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:44.662245 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:44.662211    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:45.047987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:45.047961    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"]
Oct 30 19:32:45.049625 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:32:45.049590    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82260772_1e21_46a5_bc36_4c4cc7ec8f75.slice/crio-57133b61d155148113d86d0e8a91df480cc5bed2b6aba34f326d578f478c36da WatchSource:0}: Error finding container 57133b61d155148113d86d0e8a91df480cc5bed2b6aba34f326d578f478c36da: Status 404 returned error can't find the container with id 57133b61d155148113d86d0e8a91df480cc5bed2b6aba34f326d578f478c36da
Oct 30 19:32:45.291982 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:45.291949    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f" event={"ID":"82260772-1e21-46a5-bc36-4c4cc7ec8f75","Type":"ContainerStarted","Data":"57133b61d155148113d86d0e8a91df480cc5bed2b6aba34f326d578f478c36da"}
Oct 30 19:32:45.294781 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:45.294741    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" event={"ID":"29b0996b-c55b-4e37-8220-1302701f6260","Type":"ContainerStarted","Data":"211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7"}
Oct 30 19:32:45.439121 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:45.439092    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:45.440179 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:45.440157    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:32:45.440284 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:45.440202    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 19:32:46.302331 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:46.302237    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f" event={"ID":"82260772-1e21-46a5-bc36-4c4cc7ec8f75","Type":"ContainerStarted","Data":"bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274"}
Oct 30 19:32:46.302743 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:46.302439    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:46.416427 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:46.416362    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f" podStartSLOduration=2.560193232 podStartE2EDuration="3.416348902s" totalImagesPullingTime="856.15567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:46.415562652 +0000 UTC m=+1013.626779798" watchObservedRunningTime="2026-10-30 19:32:46.416348902 +0000 UTC m=+1013.627566046"
Oct 30 19:32:46.439342 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:46.439313    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:32:46.439480 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:46.439367    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 19:32:47.220459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:47.220428    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 19:32:47.256303 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:47.256281    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 19:32:47.439808 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:47.439772    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:32:47.440212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:47.439841    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 19:32:48.439338 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:48.439304    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:32:48.439524 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:48.439358    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 19:32:49.009392 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.009364    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:49.123778 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.123749    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"]
Oct 30 19:32:49.126803 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.126786    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.179838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.179060    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-686689-serving-cert\""
Oct 30 19:32:49.232522 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.232488    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"]
Oct 30 19:32:49.295705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.295589    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-maas-api-tls\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.295705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.295658    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-metrics-certs\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.295705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.295687    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6njht\" (UniqueName: \"kubernetes.io/projected/dab3dae4-c6eb-4256-95a1-c5354500fba4-kube-api-access-6njht\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.296024 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.295724    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab3dae4-c6eb-4256-95a1-c5354500fba4-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.397115 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.397084    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-maas-api-tls\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.397355 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.397135    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-metrics-certs\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.397355 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.397162    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6njht\" (UniqueName: \"kubernetes.io/projected/dab3dae4-c6eb-4256-95a1-c5354500fba4-kube-api-access-6njht\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.397355 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.397205    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab3dae4-c6eb-4256-95a1-c5354500fba4-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.397355 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:32:49.397268    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-serving-cert: secret "maas-api-e2e-worker-w5-686689-serving-cert" not found
Oct 30 19:32:49.397355 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:32:49.397343    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-maas-api-tls podName:dab3dae4-c6eb-4256-95a1-c5354500fba4 nodeName:}" failed. No retries permitted until 2026-10-30 19:32:49.897322413 +0000 UTC m=+1017.108539560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-maas-api-tls") pod "maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g" (UID: "dab3dae4-c6eb-4256-95a1-c5354500fba4") : secret "maas-api-e2e-worker-w5-686689-serving-cert" not found
Oct 30 19:32:49.397911 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.397892    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab3dae4-c6eb-4256-95a1-c5354500fba4-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.400273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.400249    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-metrics-certs\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.438893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.438858    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 19:32:49.439052 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.438920    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 19:32:49.465448 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.465419    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6njht\" (UniqueName: \"kubernetes.io/projected/dab3dae4-c6eb-4256-95a1-c5354500fba4-kube-api-access-6njht\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.903369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.903331    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-maas-api-tls\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:49.905875 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:49.905848    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-maas-api-tls\") pod \"maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:50.038605 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:50.038576    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:50.426331 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:50.426306    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"]
Oct 30 19:32:50.428290 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:32:50.428260    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab3dae4_c6eb_4256_95a1_c5354500fba4.slice/crio-501116e5a720ede9d9e62a4b352823dd53e9353e035284ddd2fc545d22ad22e6 WatchSource:0}: Error finding container 501116e5a720ede9d9e62a4b352823dd53e9353e035284ddd2fc545d22ad22e6: Status 404 returned error can't find the container with id 501116e5a720ede9d9e62a4b352823dd53e9353e035284ddd2fc545d22ad22e6
Oct 30 19:32:50.442991 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:50.442966    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:50.443884 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:50.443866    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:32:51.328552 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:51.328518    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g" event={"ID":"dab3dae4-c6eb-4256-95a1-c5354500fba4","Type":"ContainerStarted","Data":"9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48"}
Oct 30 19:32:51.328927 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:51.328579    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g" event={"ID":"dab3dae4-c6eb-4256-95a1-c5354500fba4","Type":"ContainerStarted","Data":"501116e5a720ede9d9e62a4b352823dd53e9353e035284ddd2fc545d22ad22e6"}
Oct 30 19:32:51.328927 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:51.328610    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:32:51.456555 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:51.456496    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g" podStartSLOduration=1.654020535 podStartE2EDuration="2.456481017s" totalImagesPullingTime="802.460482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:51.455975107 +0000 UTC m=+1018.667192254" watchObservedRunningTime="2026-10-30 19:32:51.456481017 +0000 UTC m=+1018.667698163"
Oct 30 19:32:52.314180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:52.314148    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 19:32:57.341294 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:32:57.341261    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 19:33:00.099973 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:00.099929    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:33:00.100417 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:00.099935    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:33:00.100417 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:00.100014    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:00.100417 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:00.100079    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:33:02.452552 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.452521    4420 generic.go:356] "Generic (PLEG): container finished" podID="36d86524-a126-407d-85ed-5a62b852046f" containerID="59ece95686bb3dc10cbdd236182b2afdb39a93b02c77570e42cf4c8d4db7711f" exitCode=0
Oct 30 19:33:02.453054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.452592    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" event={"ID":"36d86524-a126-407d-85ed-5a62b852046f","Type":"ContainerDied","Data":"59ece95686bb3dc10cbdd236182b2afdb39a93b02c77570e42cf4c8d4db7711f"}
Oct 30 19:33:02.453136 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.453066    4420 scope.go:122] "RemoveContainer" containerID="59ece95686bb3dc10cbdd236182b2afdb39a93b02c77570e42cf4c8d4db7711f"
Oct 30 19:33:02.454312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.454290    4420 generic.go:356] "Generic (PLEG): container finished" podID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerID="161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e" exitCode=0
Oct 30 19:33:02.454390 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.454335    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" event={"ID":"a5a35842-06c7-43f1-b1b5-cdabd86db545","Type":"ContainerDied","Data":"161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e"}
Oct 30 19:33:02.454497 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.454472    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:02.454553 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.454540    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:02.454587 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.454576    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:02.454648 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.454636    4420 scope.go:122] "RemoveContainer" containerID="161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e"
Oct 30 19:33:02.455685 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.455663    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:02.455759 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.455719    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:02.455759 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:02.455750    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:03.510383 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:03.510333    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" event={"ID":"36d86524-a126-407d-85ed-5a62b852046f","Type":"ContainerStarted","Data":"5abebb31940b25d226a8714f7afbc47095350eae061d6d4afa086452cef7ce4f"}
Oct 30 19:33:03.513658 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:03.513628    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" event={"ID":"a5a35842-06c7-43f1-b1b5-cdabd86db545","Type":"ContainerStarted","Data":"c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae"}
Oct 30 19:33:03.779903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:03.776164    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:33:03.779903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:03.776296    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:33:03.779903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:03.776360    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 19:33:04.102895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:04.102179    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:33:04.102895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:04.102309    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:33:04.102895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:04.102354    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 19:33:04.774250 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:04.774219    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:33:04.774861 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:04.774804    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 19:33:05.101418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:05.101327    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:33:05.101665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:05.101636    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 19:33:05.233913 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:33:05.233884    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:33:05.778841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:05.773268    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:33:05.778841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:05.773325    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 19:33:06.101296 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:06.101210    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:33:06.101542 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:06.101513    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 19:33:06.775655 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:06.775216    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:33:06.775655 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:06.775278    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 19:33:07.105904 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:07.104585    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:33:07.105904 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:07.104663    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 19:33:07.775206 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:07.775169    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Oct 30 19:33:07.775512 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:07.775472    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Oct 30 19:33:08.101398 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:08.101313    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 30 19:33:08.101683 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:08.101656    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 30 19:33:08.776122 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:08.776093    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:33:08.776564 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:08.776384    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:33:08.777115 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:08.777098    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:33:09.102947 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:09.102866    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:33:09.103153 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:09.103136    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:33:09.103805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:09.103787    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 19:33:23.630947 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:23.630809    4420 generic.go:356] "Generic (PLEG): container finished" podID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerID="f5ea3b5ae48e1b6d63ab68f2a812ff11c23b88d287e4ee723bc0be585595e260" exitCode=0
Oct 30 19:33:23.630947 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:23.630849    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" event={"ID":"90a417e0-9ddc-4187-aaf8-632e3e20401f","Type":"ContainerDied","Data":"f5ea3b5ae48e1b6d63ab68f2a812ff11c23b88d287e4ee723bc0be585595e260"}
Oct 30 19:33:23.631389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:23.631352    4420 scope.go:122] "RemoveContainer" containerID="f5ea3b5ae48e1b6d63ab68f2a812ff11c23b88d287e4ee723bc0be585595e260"
Oct 30 19:33:23.632664 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:23.632636    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:23.632773 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:23.632713    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:23.632773 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:23.632761    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:33:24.109183 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:24.109149    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:33:24.639493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:24.639457    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" event={"ID":"90a417e0-9ddc-4187-aaf8-632e3e20401f","Type":"ContainerStarted","Data":"761902ef60a86f2d4d82b8050ca9bfd3e6563820396f99491ceaa313bc728c9d"}
Oct 30 19:33:25.098285 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:25.098210    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:33:25.099210 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:25.099192    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 19:33:25.099273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:25.099232    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 19:33:26.098276 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:26.098236    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 19:33:26.098702 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:26.098287    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 19:33:27.098335 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:27.098305    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 19:33:27.098708 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:27.098358    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 19:33:28.098649 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:28.098619    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 19:33:28.099112 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:28.098671    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 19:33:29.098606 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:29.098569    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 30 19:33:29.098790 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:29.098629    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 30 19:33:30.102498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:30.102469    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:33:30.102985 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:30.102750    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:33:30.103386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:30.103371    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:33:35.195903 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:33:35.195867    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:33:59.797268 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.797236    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"]
Oct 30 19:33:59.800314 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.800293    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.858338 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.858298    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44abbb2-14b5-4553-b03d-e29000457db2-tls-certs\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.858492 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.858455    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslt2\" (UniqueName: \"kubernetes.io/projected/c44abbb2-14b5-4553-b03d-e29000457db2-kube-api-access-xslt2\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.858547 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.858496    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-model-cache\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.858547 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.858543    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-tmp-dir\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.858631 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.858581    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-home\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.858721 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.858702    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-dshm\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.901459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.901411    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"default-dockercfg-4lbfj\""
Oct 30 19:33:59.901459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.901434    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"openshift-service-ca.crt\""
Oct 30 19:33:59.901459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.901413    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"kube-root-ca.crt\""
Oct 30 19:33:59.901699 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.901437    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2d9861\"/\"simulator-w0-2d9861-kserve-self-signed-certs\""
Oct 30 19:33:59.903981 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.903965    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"]
Oct 30 19:33:59.964185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.964147    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-home\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.964912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.964874    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-dshm\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.965194 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.965173    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44abbb2-14b5-4553-b03d-e29000457db2-tls-certs\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.965415 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.965394    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslt2\" (UniqueName: \"kubernetes.io/projected/c44abbb2-14b5-4553-b03d-e29000457db2-kube-api-access-xslt2\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.965556 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.965540    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-home\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.965903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.965544    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-model-cache\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.966040 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.965966    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-tmp-dir\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.966909 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.966886    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-tmp-dir\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.967259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.967200    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-model-cache\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.969964 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.969787    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44abbb2-14b5-4553-b03d-e29000457db2-tls-certs\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:33:59.969964 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:33:59.969854    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-dshm\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:34:00.023253 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:00.023225    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslt2\" (UniqueName: \"kubernetes.io/projected/c44abbb2-14b5-4553-b03d-e29000457db2-kube-api-access-xslt2\") pod \"simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") " pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:34:00.121694 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:00.121637    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:34:00.494250 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:00.494228    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"]
Oct 30 19:34:00.497019 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:34:00.496986    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc44abbb2_14b5_4553_b03d_e29000457db2.slice/crio-b99794922e0f9ba66ea79bbc42fe2ff28d3de1b5c03e5df1badb8309d37842e5 WatchSource:0}: Error finding container b99794922e0f9ba66ea79bbc42fe2ff28d3de1b5c03e5df1badb8309d37842e5: Status 404 returned error can't find the container with id b99794922e0f9ba66ea79bbc42fe2ff28d3de1b5c03e5df1badb8309d37842e5
Oct 30 19:34:00.843117 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:00.843026    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" event={"ID":"c44abbb2-14b5-4553-b03d-e29000457db2","Type":"ContainerStarted","Data":"9a2b97b50d737874e5045284244cd09cb1b53c0b420452f62e23c4ef2a14f074"}
Oct 30 19:34:00.843117 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:00.843078    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" event={"ID":"c44abbb2-14b5-4553-b03d-e29000457db2","Type":"ContainerStarted","Data":"b99794922e0f9ba66ea79bbc42fe2ff28d3de1b5c03e5df1badb8309d37842e5"}
Oct 30 19:34:00.960688 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:00.960637    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podStartSLOduration=1.960623387 podStartE2EDuration="1.960623387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:00.959518625 +0000 UTC m=+1088.170735769" watchObservedRunningTime="2026-10-30 19:34:00.960623387 +0000 UTC m=+1088.171840531"
Oct 30 19:34:01.122506 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:01.122422    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:34:05.194980 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:34:05.194949    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:34:10.122195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:10.122161    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:34:10.134063 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:10.134033    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:34:10.911418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:10.911392    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:34:35.195768 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:34:35.195695    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:34:41.552712 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.552676    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"]
Oct 30 19:34:41.555339 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.555322    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.603318 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.603282    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tmp-dir\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.603495 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.603398    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-dshm\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.603495 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.603431    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tls-certs\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.603495 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.603468    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-home\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.603656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.603503    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b6s5\" (UniqueName: \"kubernetes.io/projected/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-kube-api-access-8b6s5\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.603656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.603547    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-model-cache\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.656618 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.656575    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d52df0\"/\"premium-w1-d52df0-kserve-self-signed-certs\""
Oct 30 19:34:41.656618 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.656575    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d52df0\"/\"openshift-service-ca.crt\""
Oct 30 19:34:41.656618 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.656575    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d52df0\"/\"kube-root-ca.crt\""
Oct 30 19:34:41.656618 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.656612    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d52df0\"/\"default-dockercfg-9vbhz\""
Oct 30 19:34:41.660201 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.660177    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"]
Oct 30 19:34:41.705768 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.705714    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tmp-dir\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.706034 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.705861    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-dshm\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.706034 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.705893    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tls-certs\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.706034 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.705930    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-home\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.706034 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.705965    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b6s5\" (UniqueName: \"kubernetes.io/projected/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-kube-api-access-8b6s5\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.706034 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.706005    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-model-cache\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.706727 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.706700    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-model-cache\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.707060 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.707043    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tmp-dir\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.708592 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.708210    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-home\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.711988 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.711962    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-dshm\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.717322 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.717246    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tls-certs\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.766250 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.766212    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b6s5\" (UniqueName: \"kubernetes.io/projected/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-kube-api-access-8b6s5\") pod \"premium-w1-d52df0-kserve-7cd9655bdd-lmjvs\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") " pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:41.869546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:41.869483    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:42.259260 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:42.259230    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"]
Oct 30 19:34:43.069547 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:43.069512    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" event={"ID":"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299","Type":"ContainerStarted","Data":"1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227"}
Oct 30 19:34:43.069933 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:43.069559    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" event={"ID":"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299","Type":"ContainerStarted","Data":"b08292855379e289b3f03aaa1464e3f9a8d7d670d2ff54addea402e18564dc45"}
Oct 30 19:34:43.186302 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:43.186253    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podStartSLOduration=2.186239241 podStartE2EDuration="2.186239241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:43.184213667 +0000 UTC m=+1130.395430808" watchObservedRunningTime="2026-10-30 19:34:43.186239241 +0000 UTC m=+1130.397456386"
Oct 30 19:34:43.870093 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:43.870066    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:51.870108 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:51.870075    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:51.882356 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:51.882333    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:34:52.130576 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:34:52.130510    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:35:05.196269 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:35:05.196229    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:35:24.359896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.359863    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"]
Oct 30 19:35:24.362657 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.362641    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.458750 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.458706    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tls-certs\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.458960 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.458767    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-model-cache\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.458960 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.458908    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm27j\" (UniqueName: \"kubernetes.io/projected/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-kube-api-access-zm27j\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.459100 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.459084    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-dshm\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.459154 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.459108    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-home\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.459246 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.459221    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tmp-dir\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.466436 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.466413    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"kube-root-ca.crt\""
Oct 30 19:35:24.466588 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.466526    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"openshift-service-ca.crt\""
Oct 30 19:35:24.466656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.466622    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"premium-w2-a3f9b7-kserve-self-signed-certs\""
Oct 30 19:35:24.466656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.466624    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"default-dockercfg-x47rt\""
Oct 30 19:35:24.469005 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.468982    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"]
Oct 30 19:35:24.560766 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.560733    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tmp-dir\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.560966 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.560800    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tls-certs\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.560966 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.560859    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-model-cache\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.560966 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.560920    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm27j\" (UniqueName: \"kubernetes.io/projected/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-kube-api-access-zm27j\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.561075 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.560985    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-dshm\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.561075 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.561009    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-home\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.561200 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.561180    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tmp-dir\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.561257 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.561232    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-model-cache\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.561354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.561333    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-home\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.563292 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.563265    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-dshm\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.563494 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.563481    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tls-certs\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.619600 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.619545    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm27j\" (UniqueName: \"kubernetes.io/projected/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-kube-api-access-zm27j\") pod \"premium-w2-a3f9b7-kserve-85767776db-7lpdt\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") " pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:24.673992 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:24.673962    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:25.061023 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:25.060997    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"]
Oct 30 19:35:25.063225 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:35:25.063197    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6167cb2_b4ba_4b88_9ce4_fc2636006ec8.slice/crio-2bf72a1d4a30857c8a2b4157988bcec96db66552c80d397966e8e127f55de3e8 WatchSource:0}: Error finding container 2bf72a1d4a30857c8a2b4157988bcec96db66552c80d397966e8e127f55de3e8: Status 404 returned error can't find the container with id 2bf72a1d4a30857c8a2b4157988bcec96db66552c80d397966e8e127f55de3e8
Oct 30 19:35:25.291247 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:25.291204    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" event={"ID":"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8","Type":"ContainerStarted","Data":"67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6"}
Oct 30 19:35:25.291409 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:25.291253    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" event={"ID":"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8","Type":"ContainerStarted","Data":"2bf72a1d4a30857c8a2b4157988bcec96db66552c80d397966e8e127f55de3e8"}
Oct 30 19:35:25.408186 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:25.408127    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podStartSLOduration=1.408111158 podStartE2EDuration="1.408111158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:25.406921833 +0000 UTC m=+1172.618138980" watchObservedRunningTime="2026-10-30 19:35:25.408111158 +0000 UTC m=+1172.619328314"
Oct 30 19:35:25.674366 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:25.674293    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:34.674426 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:34.674396    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:34.686784 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:34.686763    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:35:35.196324 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:35:35.196291    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:35:35.365454 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:35:35.365430    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:36:05.194998 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:36:05.194917    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:36:13.559639 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:13.559594    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"]
Oct 30 19:36:13.560633 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:13.559889    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" podUID="1cc4d891-868b-425a-b272-f2a75d985419" containerName="istio-proxy" containerID="cri-o://b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc" gracePeriod=30
Oct 30 19:36:13.589201 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:13.589164    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"]
Oct 30 19:36:13.589690 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:13.589400    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" podUID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" containerName="istio-proxy" containerID="cri-o://897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c" gracePeriod=30
Oct 30 19:36:18.866362 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.866333    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:36:18.879229 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.879210    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:36:18.937086 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937066    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-kube-api-access-rft5d\" (UniqueName: \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-kube-api-access-rft5d\") pod \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") "
Oct 30 19:36:18.937207 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937092    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-istio-token\" (UniqueName: \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-istio-token\") pod \"1cc4d891-868b-425a-b272-f2a75d985419\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") "
Oct 30 19:36:18.937207 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937118    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-kube-api-access-6snq7\" (UniqueName: \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-kube-api-access-6snq7\") pod \"1cc4d891-868b-425a-b272-f2a75d985419\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") "
Oct 30 19:36:18.937207 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937139    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istiod-ca-cert\") pod \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") "
Oct 30 19:36:18.937207 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937157    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-token\" (UniqueName: \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-token\") pod \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") "
Oct 30 19:36:18.937207 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937177    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cc4d891-868b-425a-b272-f2a75d985419-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc4d891-868b-425a-b272-f2a75d985419-istiod-ca-cert\") pod \"1cc4d891-868b-425a-b272-f2a75d985419\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") "
Oct 30 19:36:18.937488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937231    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-data\") pod \"1cc4d891-868b-425a-b272-f2a75d985419\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") "
Oct 30 19:36:18.937488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937278    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-credential-socket\") pod \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") "
Oct 30 19:36:18.937488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937312    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-certs\") pod \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") "
Oct 30 19:36:18.937488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937365    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-socket\") pod \"1cc4d891-868b-425a-b272-f2a75d985419\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") "
Oct 30 19:36:18.937488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937389    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-envoy\") pod \"1cc4d891-868b-425a-b272-f2a75d985419\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") "
Oct 30 19:36:18.937488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937439    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-certs\") pod \"1cc4d891-868b-425a-b272-f2a75d985419\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") "
Oct 30 19:36:18.937779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937524    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-podinfo\") pod \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") "
Oct 30 19:36:18.937779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937530    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istiod-ca-cert" pod "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" (UID: "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:18.937779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937570    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-credential-socket\") pod \"1cc4d891-868b-425a-b272-f2a75d985419\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") "
Oct 30 19:36:18.937779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937582    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc4d891-868b-425a-b272-f2a75d985419-istiod-ca-cert" pod "1cc4d891-868b-425a-b272-f2a75d985419" (UID: "1cc4d891-868b-425a-b272-f2a75d985419"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:36:18.937779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937610    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-socket\") pod \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") "
Oct 30 19:36:18.937779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937674    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-envoy\") pod \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") "
Oct 30 19:36:18.937779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937709    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-data\") pod \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\" (UID: \"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf\") "
Oct 30 19:36:18.937779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937747    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cc4d891-868b-425a-b272-f2a75d985419-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc4d891-868b-425a-b272-f2a75d985419-istio-podinfo\") pod \"1cc4d891-868b-425a-b272-f2a75d985419\" (UID: \"1cc4d891-868b-425a-b272-f2a75d985419\") "
Oct 30 19:36:18.938215 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937782    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-certs" pod "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" (UID: "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.938215 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.937871    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-certs" pod "1cc4d891-868b-425a-b272-f2a75d985419" (UID: "1cc4d891-868b-425a-b272-f2a75d985419"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.938215 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938040    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-data" pod "1cc4d891-868b-425a-b272-f2a75d985419" (UID: "1cc4d891-868b-425a-b272-f2a75d985419"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.938215 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938057    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-socket" pod "1cc4d891-868b-425a-b272-f2a75d985419" (UID: "1cc4d891-868b-425a-b272-f2a75d985419"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.938420 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938263    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-data" pod "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" (UID: "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.938420 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938412    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:18.938526 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938433    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:18.938526 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938441    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-credential-socket" pod "1cc4d891-868b-425a-b272-f2a75d985419" (UID: "1cc4d891-868b-425a-b272-f2a75d985419"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.938526 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938449    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cc4d891-868b-425a-b272-f2a75d985419-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:18.938526 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938487    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:18.938526 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938503    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:18.938526 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938519    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:18.939000 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938531    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:18.939000 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938574    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-credential-socket" pod "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" (UID: "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.939000 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.938730    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-socket" pod "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" (UID: "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.940856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.940792    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-kube-api-access-rft5d" pod "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" (UID: "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf"). InnerVolumeSpecName "kube-api-access-rft5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:18.940969 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.940897    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-envoy" pod "1cc4d891-868b-425a-b272-f2a75d985419" (UID: "1cc4d891-868b-425a-b272-f2a75d985419"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.940969 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.940915    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-envoy" pod "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" (UID: "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:18.940969 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.940929    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-istio-token" pod "1cc4d891-868b-425a-b272-f2a75d985419" (UID: "1cc4d891-868b-425a-b272-f2a75d985419"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:18.941220 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.941195    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-kube-api-access-6snq7" pod "1cc4d891-868b-425a-b272-f2a75d985419" (UID: "1cc4d891-868b-425a-b272-f2a75d985419"). InnerVolumeSpecName "kube-api-access-6snq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:18.941310 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.941258    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-podinfo" pod "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" (UID: "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:18.941310 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.941291    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cc4d891-868b-425a-b272-f2a75d985419-istio-podinfo" pod "1cc4d891-868b-425a-b272-f2a75d985419" (UID: "1cc4d891-868b-425a-b272-f2a75d985419"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:36:18.941387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:18.941311    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-token" pod "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" (UID: "4e4ba9b8-54f5-4dab-9ca9-730994a59ebf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:19.039750 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.039716    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cc4d891-868b-425a-b272-f2a75d985419-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.039929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.039757    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rft5d\" (UniqueName: \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-kube-api-access-rft5d\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.039929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.039774    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.039929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.039788    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6snq7\" (UniqueName: \"kubernetes.io/projected/1cc4d891-868b-425a-b272-f2a75d985419-kube-api-access-6snq7\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.039929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.039801    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.039929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.039845    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.039929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.039860    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.039929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.039873    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.039929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.039886    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cc4d891-868b-425a-b272-f2a75d985419-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.039929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.039898    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.039929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.039910    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:36:19.623690 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.623654    4420 generic.go:356] "Generic (PLEG): container finished" podID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" containerID="897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c" exitCode=0
Oct 30 19:36:19.623842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.623707    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"
Oct 30 19:36:19.623842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.623734    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" event={"ID":"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf","Type":"ContainerDied","Data":"897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c"}
Oct 30 19:36:19.623842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.623792    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c" event={"ID":"4e4ba9b8-54f5-4dab-9ca9-730994a59ebf","Type":"ContainerDied","Data":"e089a1c19f50c4bca9dbc88c06c17ed53a442b0ea058727e3c9fe9c8bcd50064"}
Oct 30 19:36:19.623842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.623837    4420 scope.go:122] "RemoveContainer" containerID="897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c"
Oct 30 19:36:19.625257 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.625229    4420 generic.go:356] "Generic (PLEG): container finished" podID="1cc4d891-868b-425a-b272-f2a75d985419" containerID="b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc" exitCode=0
Oct 30 19:36:19.625371 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.625292    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"
Oct 30 19:36:19.625371 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.625303    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" event={"ID":"1cc4d891-868b-425a-b272-f2a75d985419","Type":"ContainerDied","Data":"b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc"}
Oct 30 19:36:19.625371 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.625343    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh" event={"ID":"1cc4d891-868b-425a-b272-f2a75d985419","Type":"ContainerDied","Data":"853e9d16d30cf8856fd09c42eb0e9dcc13fb0ae19ab031884f87294c6101e3dd"}
Oct 30 19:36:19.639059 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.639040    4420 scope.go:122] "RemoveContainer" containerID="897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c"
Oct 30 19:36:19.639335 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:36:19.639315    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c\": container with ID starting with 897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c not found: ID does not exist" containerID="897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c"
Oct 30 19:36:19.639382 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.639346    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c"} err="failed to get container status \"897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c\": rpc error: code = NotFound desc = could not find container \"897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c\": container with ID starting with 897ff94e9c8822c1103d1cdc75369b636c8ca064a826925284c507dcad75285c not found: ID does not exist"
Oct 30 19:36:19.639382 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.639363    4420 scope.go:122] "RemoveContainer" containerID="b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc"
Oct 30 19:36:19.649461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.649442    4420 scope.go:122] "RemoveContainer" containerID="b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc"
Oct 30 19:36:19.649735 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:36:19.649714    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc\": container with ID starting with b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc not found: ID does not exist" containerID="b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc"
Oct 30 19:36:19.649832 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.649743    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc"} err="failed to get container status \"b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc\": rpc error: code = NotFound desc = could not find container \"b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc\": container with ID starting with b10617dc0431e4eadfa3dc9b0d153e49e9a8f0566f0d93c11a78c4ccce5376dc not found: ID does not exist"
Oct 30 19:36:19.787084 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.787057    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"]
Oct 30 19:36:19.837090 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.837068    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-p9fnh"]
Oct 30 19:36:19.958629 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.958593    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"]
Oct 30 19:36:19.964399 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:19.964375    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-7kt6c"]
Oct 30 19:36:20.894936 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:20.894909    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc4d891-868b-425a-b272-f2a75d985419" path="/var/lib/kubelet/pods/1cc4d891-868b-425a-b272-f2a75d985419/volumes"
Oct 30 19:36:20.895320 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:20.895307    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" path="/var/lib/kubelet/pods/4e4ba9b8-54f5-4dab-9ca9-730994a59ebf/volumes"
Oct 30 19:36:23.473303 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.473273    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"]
Oct 30 19:36:23.473910 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.473881    4420 cpu_manager.go:414] "containerMap: removing container" podUID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" containerName="istio-proxy"
Oct 30 19:36:23.473910 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.473911    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" containerName="istio-proxy"
Oct 30 19:36:23.474087 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.473949    4420 cpu_manager.go:414] "containerMap: removing container" podUID="1cc4d891-868b-425a-b272-f2a75d985419" containerName="istio-proxy"
Oct 30 19:36:23.474087 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.473957    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc4d891-868b-425a-b272-f2a75d985419" containerName="istio-proxy"
Oct 30 19:36:23.474087 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.474083    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc4d891-868b-425a-b272-f2a75d985419" containerName="istio-proxy"
Oct 30 19:36:23.474206 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.474098    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4ba9b8-54f5-4dab-9ca9-730994a59ebf" containerName="istio-proxy"
Oct 30 19:36:23.478745 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.478729    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.580873 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.580845    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"kube-root-ca.crt\""
Oct 30 19:36:23.580873 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.580845    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"openshift-service-ca.crt\""
Oct 30 19:36:23.580873 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.580877    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"default-dockercfg-wljhm\""
Oct 30 19:36:23.581147 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.580979    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b5178f\"/\"premium-w4-b5178f-kserve-self-signed-certs\""
Oct 30 19:36:23.583106 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.583083    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"]
Oct 30 19:36:23.589250 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.589228    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-dshm\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.589352 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.589280    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7dxk\" (UniqueName: \"kubernetes.io/projected/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-kube-api-access-q7dxk\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.589352 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.589341    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-model-cache\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.589414 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.589359    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tmp-dir\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.589414 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.589383    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-home\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.589493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.589446    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tls-certs\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.690304 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.690274    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tls-certs\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.690493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.690341    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-dshm\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.690493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.690379    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7dxk\" (UniqueName: \"kubernetes.io/projected/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-kube-api-access-q7dxk\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.690493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.690402    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-model-cache\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.690632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.690533    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tmp-dir\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.690632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.690579    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-home\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.690843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.690793    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-model-cache\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.690940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.690919    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tmp-dir\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.690985 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.690954    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-home\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.692739 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.692719    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-dshm\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.692971 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.692955    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tls-certs\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.746926 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.746861    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7dxk\" (UniqueName: \"kubernetes.io/projected/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-kube-api-access-q7dxk\") pod \"premium-w4-b5178f-kserve-899fd984c-zrkmn\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") " pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:23.790663 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:23.790635    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:24.173978 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:24.173950    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"]
Oct 30 19:36:24.175920 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:36:24.175896    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a0e5afb_30a8_41e6_bc35_9c5e2f9b488b.slice/crio-f9bb244e209f097c19152beac576ff5d053b1edbc4a48bd72cadee9bfb15eece WatchSource:0}: Error finding container f9bb244e209f097c19152beac576ff5d053b1edbc4a48bd72cadee9bfb15eece: Status 404 returned error can't find the container with id f9bb244e209f097c19152beac576ff5d053b1edbc4a48bd72cadee9bfb15eece
Oct 30 19:36:24.653910 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:24.653865    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" event={"ID":"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b","Type":"ContainerStarted","Data":"ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1"}
Oct 30 19:36:24.653910 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:24.653913    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" event={"ID":"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b","Type":"ContainerStarted","Data":"f9bb244e209f097c19152beac576ff5d053b1edbc4a48bd72cadee9bfb15eece"}
Oct 30 19:36:24.790904 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:24.790879    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:24.801562 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:24.801499    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podStartSLOduration=1.8014814129999999 podStartE2EDuration="1.801481413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:24.799489396 +0000 UTC m=+1232.010706541" watchObservedRunningTime="2026-10-30 19:36:24.801481413 +0000 UTC m=+1232.012698559"
Oct 30 19:36:33.791204 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:33.791170    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:33.803379 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:33.803359    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:34.719377 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:36:34.719350    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:36:35.196090 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:36:35.196058    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:37:05.195706 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:37:05.195670    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:37:05.588132 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.584326    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"]
Oct 30 19:37:05.589970 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.589950    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.644507 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.644465    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"]
Oct 30 19:37:05.690439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.690412    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d52df0\"/\"distinct-w1-d52df0-kserve-self-signed-certs\""
Oct 30 19:37:05.710422 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.710387    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-home\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.710547 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.710472    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-model-cache\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.710609 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.710584    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7rv\" (UniqueName: \"kubernetes.io/projected/d7fa2065-d5ee-4487-b82e-021ade5b8378-kube-api-access-7d7rv\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.710654 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.710639    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-tmp-dir\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.710755 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.710741    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-dshm\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.710903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.710772    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7fa2065-d5ee-4487-b82e-021ade5b8378-tls-certs\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.812047 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.812020    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-home\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.812233 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.812085    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-model-cache\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.812233 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.812117    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7rv\" (UniqueName: \"kubernetes.io/projected/d7fa2065-d5ee-4487-b82e-021ade5b8378-kube-api-access-7d7rv\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.812233 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.812140    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-tmp-dir\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.812233 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.812169    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-dshm\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.812233 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.812191    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7fa2065-d5ee-4487-b82e-021ade5b8378-tls-certs\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.812502 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.812480    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-home\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.812544 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.812521    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-model-cache\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.812584 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.812544    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-tmp-dir\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.814539 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.814514    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-dshm\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.814666 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.814648    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7fa2065-d5ee-4487-b82e-021ade5b8378-tls-certs\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.869498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.869442    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7rv\" (UniqueName: \"kubernetes.io/projected/d7fa2065-d5ee-4487-b82e-021ade5b8378-kube-api-access-7d7rv\") pod \"distinct-w1-d52df0-kserve-756846d4bc-vvthz\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") " pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:05.901548 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:05.901521    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:06.291653 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:06.291633    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"]
Oct 30 19:37:06.294474 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:37:06.294447    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7fa2065_d5ee_4487_b82e_021ade5b8378.slice/crio-1be8a2c95d98a9af393d9feeed7378288b8aae532e13a11325745084dc101260 WatchSource:0}: Error finding container 1be8a2c95d98a9af393d9feeed7378288b8aae532e13a11325745084dc101260: Status 404 returned error can't find the container with id 1be8a2c95d98a9af393d9feeed7378288b8aae532e13a11325745084dc101260
Oct 30 19:37:06.894899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:06.894865    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" event={"ID":"d7fa2065-d5ee-4487-b82e-021ade5b8378","Type":"ContainerStarted","Data":"7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6"}
Oct 30 19:37:06.895059 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:06.894909    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" event={"ID":"d7fa2065-d5ee-4487-b82e-021ade5b8378","Type":"ContainerStarted","Data":"1be8a2c95d98a9af393d9feeed7378288b8aae532e13a11325745084dc101260"}
Oct 30 19:37:06.902218 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:06.902185    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:07.006104 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:07.006057    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podStartSLOduration=2.006043898 podStartE2EDuration="2.006043898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:07.002661261 +0000 UTC m=+1274.213878409" watchObservedRunningTime="2026-10-30 19:37:07.006043898 +0000 UTC m=+1274.217261043"
Oct 30 19:37:15.901675 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:15.901635    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:15.917966 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:15.917936    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:15.971624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:15.971593    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:37:34.920859 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:34.920777    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"]
Oct 30 19:37:34.925743 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:34.925726    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.029488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.029456    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"]
Oct 30 19:37:35.048614 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.048579    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.048741 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.048615    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.048783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.048725    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.048908 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.048893    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.048974 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.048961    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.049010 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.048986    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-266vr\" (UniqueName: \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-kube-api-access-266vr\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.049065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.049020    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.049115 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.049072    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.049158 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.049110    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.149800 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.149767    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.149800 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.149801    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266vr\" (UniqueName: \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-kube-api-access-266vr\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.149853    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.149869    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.149883    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.149911    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.149925    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.149975    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150314 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.150068    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150314 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.150243    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150421 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.150390    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150464 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.150451    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150505 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.150488    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.150727 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.150700    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.152426 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.152398    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.152586 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.152571    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.195194 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:37:35.195128    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:37:35.206322 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.206301    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.206464 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.206441    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-266vr\" (UniqueName: \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-kube-api-access-266vr\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.238443 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.238415    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:35.662792 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.662767    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"]
Oct 30 19:37:35.664349 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:37:35.664323    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfde16f9_2bea_4cf0_8c3f_7e689934de26.slice/crio-ede054ce7e5d2c5473f8084f773b409150555483f0d4fc901f123007bd279a13 WatchSource:0}: Error finding container ede054ce7e5d2c5473f8084f773b409150555483f0d4fc901f123007bd279a13: Status 404 returned error can't find the container with id ede054ce7e5d2c5473f8084f773b409150555483f0d4fc901f123007bd279a13
Oct 30 19:37:35.666664 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.666630    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:35.666718 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.666699    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:35.666752 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:35.666728    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:37:36.069769 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:36.069686    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" event={"ID":"bfde16f9-2bea-4cf0-8c3f-7e689934de26","Type":"ContainerStarted","Data":"989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1"}
Oct 30 19:37:36.069769 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:36.069742    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" event={"ID":"bfde16f9-2bea-4cf0-8c3f-7e689934de26","Type":"ContainerStarted","Data":"ede054ce7e5d2c5473f8084f773b409150555483f0d4fc901f123007bd279a13"}
Oct 30 19:37:36.187301 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:36.187239    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" podStartSLOduration=2.18721869 podStartE2EDuration="2.18721869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:36.186132295 +0000 UTC m=+1303.397349451" watchObservedRunningTime="2026-10-30 19:37:36.18721869 +0000 UTC m=+1303.398435837"
Oct 30 19:37:36.238976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:36.238946    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:36.240545 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:36.240524    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:37:36.240655 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:36.240568    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:37:37.239488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:37.239458    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:37:37.239892 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:37.239509    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:37:38.239085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:38.239055    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:37:38.239263 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:38.239099    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:37:39.238857 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:39.238803    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:37:39.239248 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:39.238874    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:37:40.188584 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.188551    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"]
Oct 30 19:37:40.192689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.192667    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.218421 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.218394    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-dshm\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.218546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.218450    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-model-cache\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.218594 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.218538    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkt44\" (UniqueName: \"kubernetes.io/projected/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-kube-api-access-lkt44\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.218594 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.218573    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-home\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.218710 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.218643    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tmp-dir\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.218765 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.218722    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tls-certs\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.238970 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.238946    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:37:40.239366 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.238992    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:37:40.270010 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.269985    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a3f9b7\"/\"distinct-w2-a3f9b7-kserve-self-signed-certs\""
Oct 30 19:37:40.296202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.296175    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"]
Oct 30 19:37:40.319402 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.319373    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-dshm\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.319544 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.319433    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-model-cache\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.319544 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.319455    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkt44\" (UniqueName: \"kubernetes.io/projected/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-kube-api-access-lkt44\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.319544 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.319473    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-home\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.319544 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.319491    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tmp-dir\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.319544 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.319532    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tls-certs\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.320245 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.319919    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-home\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.320245 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.319933    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-model-cache\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.320245 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.320015    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tmp-dir\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.321755 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.321736    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-dshm\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.322123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.322106    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tls-certs\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.386117 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.386040    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkt44\" (UniqueName: \"kubernetes.io/projected/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-kube-api-access-lkt44\") pod \"distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") " pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.505505 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.505426    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:40.893244 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:37:40.893212    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda934ee61_e753_4b2a_8fc5_2e9d84698ca3.slice/crio-31b10c627ed1d43290fa8f3274720c37690ff4cc3856fe29e7d3b411b6ffff14 WatchSource:0}: Error finding container 31b10c627ed1d43290fa8f3274720c37690ff4cc3856fe29e7d3b411b6ffff14: Status 404 returned error can't find the container with id 31b10c627ed1d43290fa8f3274720c37690ff4cc3856fe29e7d3b411b6ffff14
Oct 30 19:37:40.896189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:40.896135    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"]
Oct 30 19:37:41.119747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:41.119704    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" event={"ID":"a934ee61-e753-4b2a-8fc5-2e9d84698ca3","Type":"ContainerStarted","Data":"59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42"}
Oct 30 19:37:41.119950 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:41.119762    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" event={"ID":"a934ee61-e753-4b2a-8fc5-2e9d84698ca3","Type":"ContainerStarted","Data":"31b10c627ed1d43290fa8f3274720c37690ff4cc3856fe29e7d3b411b6ffff14"}
Oct 30 19:37:41.239212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:41.239122    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:37:41.239212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:41.239172    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:37:41.241265 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:41.241222    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podStartSLOduration=1.241208972 podStartE2EDuration="1.241208972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:41.239638049 +0000 UTC m=+1308.450855195" watchObservedRunningTime="2026-10-30 19:37:41.241208972 +0000 UTC m=+1308.452426116"
Oct 30 19:37:41.505855 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:41.505758    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:42.239640 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:42.239614    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Oct 30 19:37:42.240025 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:42.239663    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Oct 30 19:37:43.243439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:43.243412    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:43.243846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:43.243689    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:43.244278 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:43.244260    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:37:50.505929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:50.505888    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:50.517731 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:50.517696    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:51.187225 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:51.187200    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:37:59.186848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.186781    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"]
Oct 30 19:37:59.191572 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.191550    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.301712 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.301683    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"openshift-service-ca.crt\""
Oct 30 19:37:59.301962 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.301837    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"default-dockercfg-244mk\""
Oct 30 19:37:59.302118 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.301919    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"kube-root-ca.crt\""
Oct 30 19:37:59.303192 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.303173    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-686689\"/\"distinct-w5-686689-kserve-self-signed-certs\""
Oct 30 19:37:59.304527 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.304508    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"]
Oct 30 19:37:59.331755 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.331735    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tls-certs\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.331866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.331791    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-dshm\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.331916 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.331893    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-model-cache\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.332008 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.331995    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tmp-dir\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.332047 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.332029    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwmqq\" (UniqueName: \"kubernetes.io/projected/f91701fe-f82f-4367-84c5-5a5ff7355fc1-kube-api-access-rwmqq\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.332079 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.332050    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-home\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.434614 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.434563    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tls-certs\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.434843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.434679    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-dshm\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.434843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.434711    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-model-cache\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.434843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.434780    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tmp-dir\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.435044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.434851    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwmqq\" (UniqueName: \"kubernetes.io/projected/f91701fe-f82f-4367-84c5-5a5ff7355fc1-kube-api-access-rwmqq\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.435044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.434889    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-home\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.435756 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.435441    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-home\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.442178 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.442105    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tls-certs\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.442874 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.442853    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tmp-dir\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.443453 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.443433    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-model-cache\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.447028 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.447007    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-dshm\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.500635 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.500611    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwmqq\" (UniqueName: \"kubernetes.io/projected/f91701fe-f82f-4367-84c5-5a5ff7355fc1-kube-api-access-rwmqq\") pod \"distinct-w5-686689-kserve-594c55c4c7-8zvlm\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") " pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.505436 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.505413    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:37:59.887645 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:37:59.887619    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"]
Oct 30 19:37:59.890281 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:37:59.890250    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf91701fe_f82f_4367_84c5_5a5ff7355fc1.slice/crio-c03696583b260538c1d0d724214b7a9c51cabb75ad82aa59f80e1b3963f764c3 WatchSource:0}: Error finding container c03696583b260538c1d0d724214b7a9c51cabb75ad82aa59f80e1b3963f764c3: Status 404 returned error can't find the container with id c03696583b260538c1d0d724214b7a9c51cabb75ad82aa59f80e1b3963f764c3
Oct 30 19:38:00.229430 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:38:00.229396    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" event={"ID":"f91701fe-f82f-4367-84c5-5a5ff7355fc1","Type":"ContainerStarted","Data":"b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0"}
Oct 30 19:38:00.229430 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:38:00.229434    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" event={"ID":"f91701fe-f82f-4367-84c5-5a5ff7355fc1","Type":"ContainerStarted","Data":"c03696583b260538c1d0d724214b7a9c51cabb75ad82aa59f80e1b3963f764c3"}
Oct 30 19:38:00.344361 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:38:00.344317    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podStartSLOduration=1.344303352 podStartE2EDuration="1.344303352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:00.343098011 +0000 UTC m=+1327.554315157" watchObservedRunningTime="2026-10-30 19:38:00.344303352 +0000 UTC m=+1327.555520495"
Oct 30 19:38:00.505951 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:38:00.505866    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:38:05.196707 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:38:05.196669    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:38:09.505711 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:38:09.505678    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:38:09.518303 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:38:09.518279    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:38:10.292275 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:38:10.292245    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:38:35.199618 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:38:35.199582    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:39:05.196791 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:39:05.196713    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:39:20.999026 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:20.998997    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"]
Oct 30 19:39:21.002970 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.002954    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.054896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.054870    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5f55cd-openshift-default-dockercfg-wh68v\""
Oct 30 19:39:21.078261 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.078237    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.078388 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.078288    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.078388 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.078347    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.078388 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.078370    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.078528 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.078446    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.078528 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.078466    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.078605 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.078590    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.078642 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.078607    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.078642 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.078622    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlz2f\" (UniqueName: \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-kube-api-access-jlz2f\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.107090 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.107063    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"]
Oct 30 19:39:21.179710 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.179673    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.179710 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.179707    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.179948 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.179733    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlz2f\" (UniqueName: \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-kube-api-access-jlz2f\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.179948 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.179757    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.179948 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.179834    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.179948 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.179862    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.179948 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.179893    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.180207 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.179953    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.180207 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.179980    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.180310 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.180297    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.180369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.180349    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.180566 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.180540    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.180902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.180566    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.181030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.181011    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.182232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.182201    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.182425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.182407    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.236625 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.236595    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.236728 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.236712    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlz2f\" (UniqueName: \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-kube-api-access-jlz2f\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.315911 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.315856    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:21.743061 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.743037    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"]
Oct 30 19:39:21.744763 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:39:21.744739    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5aab9b7_bc7c_44f4_8ec9_b335764484bc.slice/crio-3289a04ee8c5c65961fc7c1423e73249577935a925fa56a67c1d8ec305bf0ab5 WatchSource:0}: Error finding container 3289a04ee8c5c65961fc7c1423e73249577935a925fa56a67c1d8ec305bf0ab5: Status 404 returned error can't find the container with id 3289a04ee8c5c65961fc7c1423e73249577935a925fa56a67c1d8ec305bf0ab5
Oct 30 19:39:21.746926 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.746894    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:21.747004 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.746959    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:21.747004 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:21.746988    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:22.697220 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:22.697179    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" event={"ID":"b5aab9b7-bc7c-44f4-8ec9-b335764484bc","Type":"ContainerStarted","Data":"7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75"}
Oct 30 19:39:22.697695 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:22.697235    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" event={"ID":"b5aab9b7-bc7c-44f4-8ec9-b335764484bc","Type":"ContainerStarted","Data":"3289a04ee8c5c65961fc7c1423e73249577935a925fa56a67c1d8ec305bf0ab5"}
Oct 30 19:39:23.316622 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:23.316581    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:23.317942 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:23.317923    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:39:23.318003 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:23.317968    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" podUID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:39:24.317073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:24.317042    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:39:24.317481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:24.317092    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" podUID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:39:25.200181 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.200125    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" podStartSLOduration=5.200111143 podStartE2EDuration="5.200111143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:22.880142237 +0000 UTC m=+1410.091359388" watchObservedRunningTime="2026-10-30 19:39:25.200111143 +0000 UTC m=+1412.411328287"
Oct 30 19:39:25.200591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.200576    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"]
Oct 30 19:39:25.204681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.204667    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.308669 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.308642    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"]
Oct 30 19:39:25.316220 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.316195    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:39:25.316366 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.316239    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" podUID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:39:25.328480 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.328451    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c160df8-7210-45d5-b6bb-d42ae62c5310-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.328871 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.328502    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.328871 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.328612    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.328871 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.328660    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.328871 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.328749    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.328871 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.328836    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.328871 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.328869    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.329131 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.328919    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.329131 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.329026    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wf6f\" (UniqueName: \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-kube-api-access-5wf6f\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.430333 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.430302    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.430333 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.430336    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.430554 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.430356    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.430554 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.430431    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wf6f\" (UniqueName: \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-kube-api-access-5wf6f\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.430554 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.430519    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c160df8-7210-45d5-b6bb-d42ae62c5310-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.430664 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.430636    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.430734 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.430707    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.430793 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.430768    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.430887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.430783    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.430887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.430810    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.431094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.431075    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.431303 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.431277    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.431443 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.431423    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.431913 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.431891    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c160df8-7210-45d5-b6bb-d42ae62c5310-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.432963 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.432936    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.433050 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.433029    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.487473 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.487401    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.487585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.487555    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wf6f\" (UniqueName: \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-kube-api-access-5wf6f\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.518546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.518522    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:25.988172 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.987711    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"]
Oct 30 19:39:25.991627 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:39:25.991595    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c160df8_7210_45d5_b6bb_d42ae62c5310.slice/crio-2b23a0146e8758c795537a68e0d44183e8d22801ffd4c0bcb82aa4f1b062723d WatchSource:0}: Error finding container 2b23a0146e8758c795537a68e0d44183e8d22801ffd4c0bcb82aa4f1b062723d: Status 404 returned error can't find the container with id 2b23a0146e8758c795537a68e0d44183e8d22801ffd4c0bcb82aa4f1b062723d
Oct 30 19:39:25.994159 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.994115    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:25.994259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.994190    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:25.994259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:25.994226    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:39:26.324307 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:26.319333    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:39:26.324307 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:26.319388    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" podUID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:39:26.724039 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:26.723996    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" event={"ID":"7c160df8-7210-45d5-b6bb-d42ae62c5310","Type":"ContainerStarted","Data":"7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4"}
Oct 30 19:39:26.724423 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:26.724063    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" event={"ID":"7c160df8-7210-45d5-b6bb-d42ae62c5310","Type":"ContainerStarted","Data":"2b23a0146e8758c795537a68e0d44183e8d22801ffd4c0bcb82aa4f1b062723d"}
Oct 30 19:39:26.842431 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:26.842381    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" podStartSLOduration=1.842367929 podStartE2EDuration="1.842367929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:26.839289211 +0000 UTC m=+1414.050506357" watchObservedRunningTime="2026-10-30 19:39:26.842367929 +0000 UTC m=+1414.053585073"
Oct 30 19:39:27.316176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:27.316144    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 30 19:39:27.316352 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:27.316201    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" podUID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 30 19:39:27.518969 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:27.518926    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:27.520258 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:27.520235    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:39:27.520382 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:27.520278    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" podUID="7c160df8-7210-45d5-b6bb-d42ae62c5310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:39:28.447128 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:28.447099    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:28.447541 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:28.447484    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:28.448560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:28.448540    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:39:28.519557 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:28.519526    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:39:28.519721 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:28.519576    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" podUID="7c160df8-7210-45d5-b6bb-d42ae62c5310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:39:29.519469 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:29.519438    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:39:29.519876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:29.519486    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" podUID="7c160df8-7210-45d5-b6bb-d42ae62c5310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:39:30.519642 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:30.519613    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:39:30.520028 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:30.519680    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" podUID="7c160df8-7210-45d5-b6bb-d42ae62c5310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:39:31.519699 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:31.519658    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:39:31.520242 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:31.519735    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" podUID="7c160df8-7210-45d5-b6bb-d42ae62c5310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:39:32.519097 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:32.519069    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 30 19:39:32.519248 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:32.519144    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" podUID="7c160df8-7210-45d5-b6bb-d42ae62c5310" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 30 19:39:33.755689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:33.755657    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:33.756191 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:33.756094    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:33.757013 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:39:33.756996    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:39:35.195341 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:39:35.195298    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:40:00.469535 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.469503    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"]
Oct 30 19:40:00.474426 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.474409    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.543770 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.543742    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"]
Oct 30 19:40:00.561088 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.561058    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.561236 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.561112    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.561285 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.561249    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.561325 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.561283    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.561393 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.561367    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.561428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.561411    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbzd\" (UniqueName: \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-kube-api-access-8wbzd\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.561524 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.561509    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.561570 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.561555    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.561683 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.561667    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.662369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.662322    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.662369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.662373    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.662602 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.662401    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.662602 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.662423    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.662602 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.662447    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.662602 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.662470    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbzd\" (UniqueName: \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-kube-api-access-8wbzd\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.662602 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.662513    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.662602 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.662552    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.662945 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.662672    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.662945 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.662798    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.663055 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.663000    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.663105 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.663057    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.663331 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.663308    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.663433 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.663400    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.665291 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.665271    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.665421 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.665347    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.719737 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.719658    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbzd\" (UniqueName: \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-kube-api-access-8wbzd\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.719737 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.719666    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:00.788119 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:00.788092    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:01.214039 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:01.214016    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"]
Oct 30 19:40:01.216092 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:40:01.216068    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ffc1f1_605c_4e82_ad64_cc1aee402b79.slice/crio-5f92aa8d59bf5fc1499988357b397032aff06e51dc6ce92034645e5d73113bd6 WatchSource:0}: Error finding container 5f92aa8d59bf5fc1499988357b397032aff06e51dc6ce92034645e5d73113bd6: Status 404 returned error can't find the container with id 5f92aa8d59bf5fc1499988357b397032aff06e51dc6ce92034645e5d73113bd6
Oct 30 19:40:01.218370 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:01.218339    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:01.218468 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:01.218409    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:01.218468 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:01.218441    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:01.929951 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:01.929915    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" event={"ID":"a1ffc1f1-605c-4e82-ad64-cc1aee402b79","Type":"ContainerStarted","Data":"6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a"}
Oct 30 19:40:01.930353 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:01.929975    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" event={"ID":"a1ffc1f1-605c-4e82-ad64-cc1aee402b79","Type":"ContainerStarted","Data":"5f92aa8d59bf5fc1499988357b397032aff06e51dc6ce92034645e5d73113bd6"}
Oct 30 19:40:02.100806 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:02.100748    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" podStartSLOduration=2.100729105 podStartE2EDuration="2.100729105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:02.099479754 +0000 UTC m=+1449.310696898" watchObservedRunningTime="2026-10-30 19:40:02.100729105 +0000 UTC m=+1449.311946249"
Oct 30 19:40:02.788617 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:02.788584    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:02.789995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:02.789965    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:40:02.790120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:02.790013    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:40:03.789380 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:03.789347    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:40:03.789889 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:03.789407    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:40:04.789384 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:04.789351    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:40:04.789557 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:04.789415    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:40:05.208679 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:40:05.208645    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:40:05.789481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:05.789452    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:40:05.789671 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:05.789510    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:40:06.788945 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:06.788906    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:40:06.789140 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:06.788971    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:40:07.788591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:07.788556    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 30 19:40:07.789004 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:07.788610    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 30 19:40:08.792747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:08.792722    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:08.793255 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:08.793227    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:08.793943 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:08.793925    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:40:35.196664 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:40:35.196579    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:40:45.062493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.062457    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"]
Oct 30 19:40:45.066267 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.066251    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.160448 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.160412    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8kfc\" (UniqueName: \"kubernetes.io/projected/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-kube-api-access-j8kfc\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.160605 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.160546    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-dshm\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.160605 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.160586    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-model-cache\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.160687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.160620    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tls-certs\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.160729 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.160703    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-home\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.160828 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.160799    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tmp-dir\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.167964 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.167941    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"default-dockercfg-7ntd4\""
Oct 30 19:40:45.168094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.167977    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"distinct-w3-ada8d7-kserve-self-signed-certs\""
Oct 30 19:40:45.168094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.168016    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"kube-root-ca.crt\""
Oct 30 19:40:45.168094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.168034    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ada8d7\"/\"openshift-service-ca.crt\""
Oct 30 19:40:45.171195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.171179    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"]
Oct 30 19:40:45.262389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.262360    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-dshm\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.262588 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.262411    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-model-cache\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.262588 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.262438    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tls-certs\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.262588 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.262463    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-home\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.262588 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.262498    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tmp-dir\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.262588 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.262543    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8kfc\" (UniqueName: \"kubernetes.io/projected/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-kube-api-access-j8kfc\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.262975 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.262942    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-home\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.263106 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.262995    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-model-cache\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.263106 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.263024    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tmp-dir\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.264976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.264952    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-dshm\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.265142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.265126    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tls-certs\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.322298 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.322237    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8kfc\" (UniqueName: \"kubernetes.io/projected/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-kube-api-access-j8kfc\") pod \"distinct-w3-ada8d7-kserve-7f97995f55-kpgwt\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") " pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.379586 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.379560    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:45.770262 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:45.770238    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"]
Oct 30 19:40:45.771365 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:40:45.771334    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb23f3a_c315_4352_9e20_dfaa96cbe3c2.slice/crio-59c448dcbe82c68e14dfdb7879ea3ae7897f0536d1a9c33c0c727f1059bb637a WatchSource:0}: Error finding container 59c448dcbe82c68e14dfdb7879ea3ae7897f0536d1a9c33c0c727f1059bb637a: Status 404 returned error can't find the container with id 59c448dcbe82c68e14dfdb7879ea3ae7897f0536d1a9c33c0c727f1059bb637a
Oct 30 19:40:46.193486 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:46.193447    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" event={"ID":"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2","Type":"ContainerStarted","Data":"6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac"}
Oct 30 19:40:46.193884 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:46.193497    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" event={"ID":"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2","Type":"ContainerStarted","Data":"59c448dcbe82c68e14dfdb7879ea3ae7897f0536d1a9c33c0c727f1059bb637a"}
Oct 30 19:40:46.310098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:46.310053    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podStartSLOduration=1.310039409 podStartE2EDuration="1.310039409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:46.307697927 +0000 UTC m=+1493.518915069" watchObservedRunningTime="2026-10-30 19:40:46.310039409 +0000 UTC m=+1493.521256553"
Oct 30 19:40:46.380558 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:46.380528    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:55.379999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:55.379967    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:55.391975 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:55.391947    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:55.902809 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:55.902778    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"]
Oct 30 19:40:55.906990 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:55.906971    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.005508 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.005472    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.005695 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.005564    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.005695 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.005596    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.005695 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.005665    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5xxj\" (UniqueName: \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-kube-api-access-q5xxj\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.005959 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.005785    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.005959 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.005857    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.006068 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.005993    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.006068 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.006035    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.006166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.006078    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.011131 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.011110    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"]
Oct 30 19:40:56.107635 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.107606    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.107795 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.107643    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.107795 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.107673    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.107795 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.107749    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.107795 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.107780    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.108059 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.107806    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.108059 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.107870    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5xxj\" (UniqueName: \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-kube-api-access-q5xxj\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.108059 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.107924    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.108059 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.107947    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.108273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.108095    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.108273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.108211    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.108273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.108240    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.108384 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.108310    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.108541 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.108522    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.110277 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.110254    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.110327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.110297    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.163842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.163753    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.164094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.164077    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5xxj\" (UniqueName: \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-kube-api-access-q5xxj\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.221392 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.221371    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:56.265147 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.265119    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:40:56.648328 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.648305    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"]
Oct 30 19:40:56.650030 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:40:56.650002    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3751f391_d63e_489b_9ae9_8ad55dd4f1b2.slice/crio-1fcff1c8ed7894d19d4f98566cf583d8269a427f9b870fa9e4dde3e3a2d3df35 WatchSource:0}: Error finding container 1fcff1c8ed7894d19d4f98566cf583d8269a427f9b870fa9e4dde3e3a2d3df35: Status 404 returned error can't find the container with id 1fcff1c8ed7894d19d4f98566cf583d8269a427f9b870fa9e4dde3e3a2d3df35
Oct 30 19:40:56.652555 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.652522    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:56.652638 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.652586    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:56.652638 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:56.652614    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:40:57.286133 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:57.286096    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" event={"ID":"3751f391-d63e-489b-9ae9-8ad55dd4f1b2","Type":"ContainerStarted","Data":"f74af44b3e78ee7bab8d0e0be426f0bf4e76df8f6c6be2e238cff2fc86dfb9bc"}
Oct 30 19:40:57.286362 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:57.286165    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" event={"ID":"3751f391-d63e-489b-9ae9-8ad55dd4f1b2","Type":"ContainerStarted","Data":"1fcff1c8ed7894d19d4f98566cf583d8269a427f9b870fa9e4dde3e3a2d3df35"}
Oct 30 19:40:57.456468 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:57.456413    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" podStartSLOduration=2.456398135 podStartE2EDuration="2.456398135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:57.454750161 +0000 UTC m=+1504.665967307" watchObservedRunningTime="2026-10-30 19:40:57.456398135 +0000 UTC m=+1504.667615345"
Oct 30 19:40:58.221530 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:58.221498    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:40:58.223600 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:58.223576    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:40:58.223740 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:58.223629    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" podUID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:40:59.222239 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:59.222206    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:40:59.222624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:40:59.222267    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" podUID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:41:00.222419 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:00.222373    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:41:00.222907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:00.222428    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" podUID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:41:00.823747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:00.823704    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"]
Oct 30 19:41:00.824099 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:00.824047    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerName="istio-proxy" containerID="cri-o://c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac" gracePeriod=30
Oct 30 19:41:01.222243 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:01.222213    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:41:01.222433 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:01.222268    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" podUID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:41:02.221980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:02.221945    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:41:02.222425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:02.221991    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" podUID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:41:03.222676 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:03.222640    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 30 19:41:03.223108 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:03.222692    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" podUID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 30 19:41:04.227660 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:04.227630    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:41:04.228156 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:04.228032    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:41:04.228750 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:04.228733    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:41:04.886692 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:04.886662    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-zhkqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:41:04.886855 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:04.886711    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:41:05.197761 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:41:05.197727    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:41:05.786382 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.786345    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"]
Oct 30 19:41:05.790637 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.790614    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:05.842979 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.842957    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-686689-openshift-default-dockercfg-75mcj\""
Oct 30 19:41:05.895185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.895159    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"]
Oct 30 19:41:05.949399 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.949367    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:05.950232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.949413    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86rpx\" (UniqueName: \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-kube-api-access-86rpx\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:05.950232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.949486    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:05.950232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.949522    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:05.950232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.949565    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:05.950232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.949610    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:05.950232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.949728    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:05.950232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.949755    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:05.950232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:05.949792    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.051480 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.051410    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.051480 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.051443    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.051480 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.051467    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.051715 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.051558    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.051715 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.051585    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rpx\" (UniqueName: \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-kube-api-access-86rpx\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.051715 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.051646    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.051715 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.051677    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.052002 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.051715    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.052002 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.051762    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.052002 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.051859    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.052220 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.052195    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.052284 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.052216    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.052284 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.052264    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.052391 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.052373    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.054139 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.054122    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.054694 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.054676    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.085779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.085758    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:41:06.108350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.108327    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.108558 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.108536    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rpx\" (UniqueName: \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-kube-api-access-86rpx\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.253927 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.253897    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-socket\") pod \"a4d98efd-3b11-4464-bec1-5b46eec5d871\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") "
Oct 30 19:41:06.254104 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254009    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-data\") pod \"a4d98efd-3b11-4464-bec1-5b46eec5d871\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") "
Oct 30 19:41:06.254104 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254079    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-podinfo\") pod \"a4d98efd-3b11-4464-bec1-5b46eec5d871\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") "
Oct 30 19:41:06.254239 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254184    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-socket" pod "a4d98efd-3b11-4464-bec1-5b46eec5d871" (UID: "a4d98efd-3b11-4464-bec1-5b46eec5d871"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:06.254322 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254299    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-envoy\") pod \"a4d98efd-3b11-4464-bec1-5b46eec5d871\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") "
Oct 30 19:41:06.254386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254310    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-data" pod "a4d98efd-3b11-4464-bec1-5b46eec5d871" (UID: "a4d98efd-3b11-4464-bec1-5b46eec5d871"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:06.254386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254358    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-token\" (UniqueName: \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-token\") pod \"a4d98efd-3b11-4464-bec1-5b46eec5d871\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") "
Oct 30 19:41:06.254496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254385    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-credential-socket\") pod \"a4d98efd-3b11-4464-bec1-5b46eec5d871\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") "
Oct 30 19:41:06.254496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254478    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-certs\") pod \"a4d98efd-3b11-4464-bec1-5b46eec5d871\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") "
Oct 30 19:41:06.254596 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254516    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4d98efd-3b11-4464-bec1-5b46eec5d871-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d98efd-3b11-4464-bec1-5b46eec5d871-istiod-ca-cert\") pod \"a4d98efd-3b11-4464-bec1-5b46eec5d871\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") "
Oct 30 19:41:06.254650 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254595    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-kube-api-access-dx745\" (UniqueName: \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-kube-api-access-dx745\") pod \"a4d98efd-3b11-4464-bec1-5b46eec5d871\" (UID: \"a4d98efd-3b11-4464-bec1-5b46eec5d871\") "
Oct 30 19:41:06.255369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254879    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-credential-socket" pod "a4d98efd-3b11-4464-bec1-5b46eec5d871" (UID: "a4d98efd-3b11-4464-bec1-5b46eec5d871"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:06.255369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.254897    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-certs" pod "a4d98efd-3b11-4464-bec1-5b46eec5d871" (UID: "a4d98efd-3b11-4464-bec1-5b46eec5d871"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:06.255369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.255220    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:06.255369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.255238    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:06.255369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.255252    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:06.255369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.255266    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:06.255369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.255337    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d98efd-3b11-4464-bec1-5b46eec5d871-istiod-ca-cert" pod "a4d98efd-3b11-4464-bec1-5b46eec5d871" (UID: "a4d98efd-3b11-4464-bec1-5b46eec5d871"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:41:06.256777 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.256756    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-envoy" pod "a4d98efd-3b11-4464-bec1-5b46eec5d871" (UID: "a4d98efd-3b11-4464-bec1-5b46eec5d871"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:41:06.256923 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.256776    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-podinfo" pod "a4d98efd-3b11-4464-bec1-5b46eec5d871" (UID: "a4d98efd-3b11-4464-bec1-5b46eec5d871"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:41:06.257383 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.257357    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-token" pod "a4d98efd-3b11-4464-bec1-5b46eec5d871" (UID: "a4d98efd-3b11-4464-bec1-5b46eec5d871"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:06.257498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.257393    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-kube-api-access-dx745" pod "a4d98efd-3b11-4464-bec1-5b46eec5d871" (UID: "a4d98efd-3b11-4464-bec1-5b46eec5d871"). InnerVolumeSpecName "kube-api-access-dx745". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:41:06.339756 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.339674    4420 generic.go:356] "Generic (PLEG): container finished" podID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerID="c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac" exitCode=0
Oct 30 19:41:06.339756 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.339727    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"
Oct 30 19:41:06.339756 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.339728    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" event={"ID":"a4d98efd-3b11-4464-bec1-5b46eec5d871","Type":"ContainerDied","Data":"c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac"}
Oct 30 19:41:06.340019 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.339792    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr" event={"ID":"a4d98efd-3b11-4464-bec1-5b46eec5d871","Type":"ContainerDied","Data":"5cb2dc63aa0f1869daa91b918d13c882ebe2d7fca2eb7e3d5a3371312c2b6b56"}
Oct 30 19:41:06.340019 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.339853    4420 scope.go:122] "RemoveContainer" containerID="c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac"
Oct 30 19:41:06.350411 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.350395    4420 scope.go:122] "RemoveContainer" containerID="c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac"
Oct 30 19:41:06.350673 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:41:06.350656    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac\": container with ID starting with c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac not found: ID does not exist" containerID="c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac"
Oct 30 19:41:06.350712 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.350688    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac"} err="failed to get container status \"c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac\": rpc error: code = NotFound desc = could not find container \"c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac\": container with ID starting with c5f35d8b4d8dcfed60d27f820c760f9ca6026e9e887fabd8a28ff28275103bac not found: ID does not exist"
Oct 30 19:41:06.356601 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.356573    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:06.356601 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.356597    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:06.356601 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.356605    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:06.356804 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.356614    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4d98efd-3b11-4464-bec1-5b46eec5d871-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:06.356804 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.356623    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx745\" (UniqueName: \"kubernetes.io/projected/a4d98efd-3b11-4464-bec1-5b46eec5d871-kube-api-access-dx745\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:41:06.404617 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.404595    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:06.507061 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.507033    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"]
Oct 30 19:41:06.511693 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.511667    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-zhkqr"]
Oct 30 19:41:06.831935 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.831906    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"]
Oct 30 19:41:06.833256 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:41:06.833230    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fe3925_27ad_4e80_be15_e6d2e4fa826e.slice/crio-d82f388cc6d575fe84418cbfa39a7602ebc40d18e0e04994b66dc11f4d008a1e WatchSource:0}: Error finding container d82f388cc6d575fe84418cbfa39a7602ebc40d18e0e04994b66dc11f4d008a1e: Status 404 returned error can't find the container with id d82f388cc6d575fe84418cbfa39a7602ebc40d18e0e04994b66dc11f4d008a1e
Oct 30 19:41:06.835543 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.835509    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:06.835654 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.835586    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:06.835654 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.835628    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:06.902399 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:06.898452    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" path="/var/lib/kubelet/pods/a4d98efd-3b11-4464-bec1-5b46eec5d871/volumes"
Oct 30 19:41:07.347388 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:07.347357    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" event={"ID":"13fe3925-27ad-4e80-be15-e6d2e4fa826e","Type":"ContainerStarted","Data":"e4b358af27be27af50da7e71f7591dc32213bf18ddeee0c902746abba543ccbc"}
Oct 30 19:41:07.347569 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:07.347412    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" event={"ID":"13fe3925-27ad-4e80-be15-e6d2e4fa826e","Type":"ContainerStarted","Data":"d82f388cc6d575fe84418cbfa39a7602ebc40d18e0e04994b66dc11f4d008a1e"}
Oct 30 19:41:07.405496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:07.405451    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:07.406561 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:07.406537    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:41:07.406676 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:07.406585    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:41:07.469015 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:07.468970    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podStartSLOduration=2.468957397 podStartE2EDuration="2.468957397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:07.466128071 +0000 UTC m=+1514.677345217" watchObservedRunningTime="2026-10-30 19:41:07.468957397 +0000 UTC m=+1514.680174585"
Oct 30 19:41:08.405558 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:08.405527    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:41:08.406001 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:08.405581    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:41:09.405672 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:09.405637    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:41:09.406083 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:09.405702    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:41:10.404998 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:10.404967    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:41:10.405172 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:10.405015    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:41:11.405704 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:11.405669    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:41:11.406111 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:11.405726    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:41:12.409907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:12.409871    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:41:12.410535 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:12.410503    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:41:13.407202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:13.407159    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:41:13.407497 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:13.407465    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:41:14.407160 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:14.407123    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 19:41:14.407755 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:14.407722    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 19:41:16.445966 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:16.445930    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:41:16.446370 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:16.445990    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:41:16.760111 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:16.760032    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:16.760524 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:16.760504    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:16.761283 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:16.761268    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:41:21.654703 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.654670    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"]
Oct 30 19:41:21.655237 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.655222    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerName="istio-proxy"
Oct 30 19:41:21.655279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.655238    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerName="istio-proxy"
Oct 30 19:41:21.655361 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.655352    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4d98efd-3b11-4464-bec1-5b46eec5d871" containerName="istio-proxy"
Oct 30 19:41:21.658978 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.658961    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.703761 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.703733    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"]
Oct 30 19:41:21.707717 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.707701    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.763510 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.763482    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"]
Oct 30 19:41:21.809499 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.809473    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgr2m\" (UniqueName: \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-kube-api-access-zgr2m\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.809658 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.809505    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.809658 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.809552    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.809779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.809670    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9813d487-1f24-47f9-9cea-47e54083d91e-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.809779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.809737    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.810083 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.809842    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.810083 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.809961    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.810083 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.809997    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.810389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.810098    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.810389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.810124    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnl79\" (UniqueName: \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-kube-api-access-mnl79\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.810389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.810165    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.810389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.810189    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.810389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.810220    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9813d487-1f24-47f9-9cea-47e54083d91e-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.810389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.810312    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.810389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.810336    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.810389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.810359    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.811705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.810410    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.811705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.810442    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.814368 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.814345    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"]
Oct 30 19:41:21.912355 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912281    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.912486 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912404    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgr2m\" (UniqueName: \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-kube-api-access-zgr2m\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.912486 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912432    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.912611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912486    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.912611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912522    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9813d487-1f24-47f9-9cea-47e54083d91e-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.912611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912565    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.912611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912590    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.912783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912625    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.912783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912650    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.912783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912717    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.912783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912743    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnl79\" (UniqueName: \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-kube-api-access-mnl79\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.912783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912772    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.913085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912796    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.913085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912847    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9813d487-1f24-47f9-9cea-47e54083d91e-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.913085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912951    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.913085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.912975    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.913085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.913001    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.913085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.913002    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.913085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.913078    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.913434 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.913080    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.913434 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.913217    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.913434 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.913319    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9813d487-1f24-47f9-9cea-47e54083d91e-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.913592 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.913431    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.913592 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.913448    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-credential-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.913592 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.913567    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-data\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.913749 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.913598    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-socket\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.913749 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.913704    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-certs\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.914432 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.914405    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istiod-ca-cert\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.915375 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.915344    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.915483 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.915407    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-envoy\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.915793 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.915771    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.916134 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.916117    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9813d487-1f24-47f9-9cea-47e54083d91e-istio-podinfo\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.968999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.968972    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgr2m\" (UniqueName: \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-kube-api-access-zgr2m\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.969208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.969189    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.969338 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.969318    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-token\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:21.969880 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.969859    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnl79\" (UniqueName: \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-kube-api-access-mnl79\") pod \"e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") " pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:21.972318 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:21.972302    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:22.023758 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.023691    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:22.397647 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.397624    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"]
Oct 30 19:41:22.398952 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:41:22.398927    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0b55de_ce41_492a_846d_bd4533cfa6cc.slice/crio-580781aeb4627a38e1fe9d3b076a4ebe38199062a9f45b7365aa5d1e23b88f12 WatchSource:0}: Error finding container 580781aeb4627a38e1fe9d3b076a4ebe38199062a9f45b7365aa5d1e23b88f12: Status 404 returned error can't find the container with id 580781aeb4627a38e1fe9d3b076a4ebe38199062a9f45b7365aa5d1e23b88f12
Oct 30 19:41:22.401346 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.401314    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:22.401427 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.401376    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:22.401427 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.401404    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:22.452861 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.452832    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"]
Oct 30 19:41:22.453592 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:41:22.453569    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9813d487_1f24_47f9_9cea_47e54083d91e.slice/crio-6ce740186cfc85ba041fa9bd5f52d3adff33da0a9e6af28d9e00b5a5668b7b0a WatchSource:0}: Error finding container 6ce740186cfc85ba041fa9bd5f52d3adff33da0a9e6af28d9e00b5a5668b7b0a: Status 404 returned error can't find the container with id 6ce740186cfc85ba041fa9bd5f52d3adff33da0a9e6af28d9e00b5a5668b7b0a
Oct 30 19:41:22.456296 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.456268    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:22.456370 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.456331    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:22.456370 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.456362    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:22.507350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.507286    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" event={"ID":"9813d487-1f24-47f9-9cea-47e54083d91e","Type":"ContainerStarted","Data":"6ce740186cfc85ba041fa9bd5f52d3adff33da0a9e6af28d9e00b5a5668b7b0a"}
Oct 30 19:41:22.509480 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.509455    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" event={"ID":"8e0b55de-ce41-492a-846d-bd4533cfa6cc","Type":"ContainerStarted","Data":"a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd"}
Oct 30 19:41:22.509611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.509505    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" event={"ID":"8e0b55de-ce41-492a-846d-bd4533cfa6cc","Type":"ContainerStarted","Data":"580781aeb4627a38e1fe9d3b076a4ebe38199062a9f45b7365aa5d1e23b88f12"}
Oct 30 19:41:22.629126 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.629075    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" podStartSLOduration=1.629061093 podStartE2EDuration="1.629061093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:22.626084794 +0000 UTC m=+1529.837301941" watchObservedRunningTime="2026-10-30 19:41:22.629061093 +0000 UTC m=+1529.840278237"
Oct 30 19:41:22.980840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:22.977910    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:23.518268 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:23.518234    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" event={"ID":"9813d487-1f24-47f9-9cea-47e54083d91e","Type":"ContainerStarted","Data":"6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb"}
Oct 30 19:41:23.741428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:23.741373    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" podStartSLOduration=2.7413571169999997 podStartE2EDuration="2.741357117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:23.690303655 +0000 UTC m=+1530.901520800" watchObservedRunningTime="2026-10-30 19:41:23.741357117 +0000 UTC m=+1530.952574261"
Oct 30 19:41:23.976876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:23.976845    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:41:23.977165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:23.977137    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:41:24.024309 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:24.024251    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:24.025790 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:24.025766    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:41:24.025930 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:24.025843    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:41:24.986846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:24.972886    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:41:24.986846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:24.972944    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:41:25.024951 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:25.024920    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:41:25.025320 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:25.024964    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:41:25.983843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:25.981680    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:41:25.983843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:25.981752    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:41:26.025240 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:26.025206    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:41:26.025611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:26.025269    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:41:26.982841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:26.982136    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:41:26.982841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:26.982217    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:41:27.025196 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:27.025166    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:41:27.025362 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:27.025225    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:41:27.981223 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:27.981186    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:41:27.981532 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:27.981501    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:41:28.024984 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:28.024956    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:41:28.025143 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:28.025001    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:41:28.972720 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:28.972692    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 30 19:41:28.973212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:28.972742    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 30 19:41:29.025143 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:29.025111    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 19:41:29.025310 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:29.025166    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 19:41:29.977371 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:29.977345    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:29.977838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:29.977802    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:29.978455 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:29.978440    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:41:30.077767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:30.077739    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:30.078780 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:30.078759    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:30.079795 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:30.079775    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:41:35.321052 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:41:35.321011    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:41:46.224842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.224792    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"]
Oct 30 19:41:46.228786 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.228769    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.297348 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.297323    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.297476 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.297360    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.297476 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.297458    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.297563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.297509    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.297691 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.297665    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8zlx\" (UniqueName: \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-kube-api-access-q8zlx\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.297771 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.297723    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.297838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.297783    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.298011 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.297986    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.298099 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.298035    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.399411 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.399383    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.399577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.399461    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.399577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.399482    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.399577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.399521    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.399577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.399549    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.399770 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.399584    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.399770 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.399620    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.399770 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.399724    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8zlx\" (UniqueName: \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-kube-api-access-q8zlx\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.399770 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.399765    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.400027 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.399968    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.400327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.400247    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.400327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.400259    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.400327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.400325    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.400327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.400342    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.401999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.401979    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.402549 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.402528    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.435681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.435655    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"]
Oct 30 19:41:46.459408 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.459378    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.459572 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.459536    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8zlx\" (UniqueName: \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-kube-api-access-q8zlx\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.544313 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.544226    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:46.969547 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.969521    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"]
Oct 30 19:41:46.970871 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:41:46.970842    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570a77ab_2b41_4cb5_8b6d_5231cf2f7c53.slice/crio-cc350211db3598792b983511e9ce027ecdf2d546d49bcde1bbf65b78cde0409c WatchSource:0}: Error finding container cc350211db3598792b983511e9ce027ecdf2d546d49bcde1bbf65b78cde0409c: Status 404 returned error can't find the container with id cc350211db3598792b983511e9ce027ecdf2d546d49bcde1bbf65b78cde0409c
Oct 30 19:41:46.973705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.973677    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:46.973792 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.973738    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:46.973792 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:46.973769    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:41:47.703513 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:47.703474    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" event={"ID":"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53","Type":"ContainerStarted","Data":"030425a44b958edbe52de0ed98d45e513a783ce7e76ae2aaf8b91fdf22889a0e"}
Oct 30 19:41:47.704023 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:47.703568    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" event={"ID":"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53","Type":"ContainerStarted","Data":"cc350211db3598792b983511e9ce027ecdf2d546d49bcde1bbf65b78cde0409c"}
Oct 30 19:41:47.878839 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:47.878596    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" podStartSLOduration=1.8785762 podStartE2EDuration="1.8785762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:47.877197107 +0000 UTC m=+1555.088414254" watchObservedRunningTime="2026-10-30 19:41:47.8785762 +0000 UTC m=+1555.089793345"
Oct 30 19:41:48.545393 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:48.545360    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:48.546982 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:48.546958    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:41:48.547105 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:48.547029    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:41:49.544879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:49.544850    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:41:49.545259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:49.544896    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:41:50.545189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:50.545147    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:41:50.545662 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:50.545249    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:41:51.545087 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:51.545059    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:41:51.545258 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:51.545106    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:41:52.545284 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:52.545250    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:41:52.545681 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:52.545316    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:41:53.545525 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:53.545488    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:41:53.546016 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:53.545545    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:41:54.549401 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:54.549369    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:54.549879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:54.549671    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:41:54.550384 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:41:54.550366    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:42:03.713977 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:03.713910    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:42:03.950988 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:03.950933    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:03.972900 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:03.971899    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:04.125851 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:04.122907    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:04.181084 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:04.180105    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:42:04.278930 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:04.272889    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:42:04.285489 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:04.284956    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:04.360704 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:04.359956    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:04.715963 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:04.715909    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:04.716948 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:04.716924    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:42:04.793010 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:04.792966    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:04.998840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:04.998716    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:42:05.007234 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.002187    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:42:05.035203 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.035174    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:42:05.035492 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.035476    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:42:05.125318 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.125261    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:05.186992 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.132279    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:42:05.199878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.188659    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:05.199878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.189167    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:42:05.358912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.356144    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:42:05.358912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.357184    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:42:05.385036 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.384977    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:05.385753 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.385730    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:42:05.677898 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.455059    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:42:05.687050 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.687021    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:42:05.733842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.682223    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:05.835024 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.539120    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:42:05.869844 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.737907    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:05.887042 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:05.850476    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:42:06.168950 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.168890    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:42:06.208016 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.207960    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:06.378883 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.378018    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:06.385488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.385453    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:06.424901 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.424070    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:06.470695 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:42:06.468020    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:42:06.470695 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.469483    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:42:06.470695 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.469630    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:06.590259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.589541    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:42:06.590259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.589597    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:06.770887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.770323    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:42:06.798128 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.791248    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:42:06.798128 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.793277    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:42:06.798128 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.797799    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:42:06.799056 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.799032    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:42:06.813702 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:06.813678    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:42:15.283675 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.283647    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"]
Oct 30 19:42:15.287562 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.287543    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.391132 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.391095    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"default-dockercfg-z7q6k\""
Oct 30 19:42:15.391132 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.391095    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"distinct-2-w6-5f55cd-kserve-self-signed-certs\""
Oct 30 19:42:15.391362 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.391103    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"openshift-service-ca.crt\""
Oct 30 19:42:15.391362 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.391302    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f55cd\"/\"kube-root-ca.crt\""
Oct 30 19:42:15.393994 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.393967    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"]
Oct 30 19:42:15.447281 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.447231    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-model-cache\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.447622 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.447591    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db68fab2-eabb-414a-b123-9d5c6f61ee72-tls-certs\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.448275 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.448250    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-dshm\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.448515 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.448491    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-home\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.448748 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.448724    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqcjs\" (UniqueName: \"kubernetes.io/projected/db68fab2-eabb-414a-b123-9d5c6f61ee72-kube-api-access-jqcjs\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.448918 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.448774    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-tmp-dir\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.550085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.550009    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-model-cache\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.550085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.550051    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db68fab2-eabb-414a-b123-9d5c6f61ee72-tls-certs\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.550323 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.550104    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-dshm\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.550323 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.550140    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-home\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.550323 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.550212    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqcjs\" (UniqueName: \"kubernetes.io/projected/db68fab2-eabb-414a-b123-9d5c6f61ee72-kube-api-access-jqcjs\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.550323 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.550246    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-tmp-dir\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.550527 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.550431    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-model-cache\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.550595 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.550577    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-home\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.550658 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.550641    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-tmp-dir\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.552380 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.552359    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-dshm\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.552725 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.552708    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db68fab2-eabb-414a-b123-9d5c6f61ee72-tls-certs\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.607454 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.607427    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqcjs\" (UniqueName: \"kubernetes.io/projected/db68fab2-eabb-414a-b123-9d5c6f61ee72-kube-api-access-jqcjs\") pod \"distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") " pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:15.899866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:15.899834    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:16.279882 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:16.279853    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"]
Oct 30 19:42:16.281423 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:42:16.281401    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb68fab2_eabb_414a_b123_9d5c6f61ee72.slice/crio-b8ac54ccd18db170d1305c9d603aa166311f3f0f1be64b32c3846c5b4905876c WatchSource:0}: Error finding container b8ac54ccd18db170d1305c9d603aa166311f3f0f1be64b32c3846c5b4905876c: Status 404 returned error can't find the container with id b8ac54ccd18db170d1305c9d603aa166311f3f0f1be64b32c3846c5b4905876c
Oct 30 19:42:16.358186 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:16.358144    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" event={"ID":"db68fab2-eabb-414a-b123-9d5c6f61ee72","Type":"ContainerStarted","Data":"b8ac54ccd18db170d1305c9d603aa166311f3f0f1be64b32c3846c5b4905876c"}
Oct 30 19:42:17.366226 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:17.366185    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" event={"ID":"db68fab2-eabb-414a-b123-9d5c6f61ee72","Type":"ContainerStarted","Data":"a2691a4a171acfdf4ba92336f0e0dcdefeb951fb2a95b1879c5369d9ca420fc3"}
Oct 30 19:42:17.481931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:17.481883    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podStartSLOduration=2.481871044 podStartE2EDuration="2.481871044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:17.479178249 +0000 UTC m=+1584.690395395" watchObservedRunningTime="2026-10-30 19:42:17.481871044 +0000 UTC m=+1584.693088188"
Oct 30 19:42:17.900262 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:17.900231    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:21.449973 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.449938    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"]
Oct 30 19:42:21.454180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.454163    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.518586 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.518557    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"]
Oct 30 19:42:21.618249 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.618210    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djxhs\" (UniqueName: \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-kube-api-access-djxhs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.618428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.618276    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.618428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.618386    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.618546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.618431    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.618546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.618505    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.618546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.618531    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.618656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.618554    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.618693 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.618661    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.618725 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.618711    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.720832 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.720732    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djxhs\" (UniqueName: \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-kube-api-access-djxhs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.720832 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.720796    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.720881    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.720916    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.720953    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721248 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.721067    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721248 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.721113    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721248 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.721202    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721411 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.721245    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721411 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.721315    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721411 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.721334    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.721622    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.721647    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.721804 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.721736    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.724042 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.724016    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.724361 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.724342    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.777495 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.777441    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:21.777919 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:21.777893    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djxhs\" (UniqueName: \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-kube-api-access-djxhs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:22.069459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:22.069387    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:22.503720 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:22.503689    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"]
Oct 30 19:42:22.543236 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:42:22.543209    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2a09c8_fe31_46a9_9cf3_1c2b74d45b85.slice/crio-e6fa39f39cb7251caffb9fc3b8d098197c6d7683b1503960bc859b8c2d333887 WatchSource:0}: Error finding container e6fa39f39cb7251caffb9fc3b8d098197c6d7683b1503960bc859b8c2d333887: Status 404 returned error can't find the container with id e6fa39f39cb7251caffb9fc3b8d098197c6d7683b1503960bc859b8c2d333887
Oct 30 19:42:22.545714 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:22.545684    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:22.545803 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:22.545752    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:22.545803 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:22.545787    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:23.416216 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:23.416182    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" event={"ID":"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85","Type":"ContainerStarted","Data":"a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab"}
Oct 30 19:42:23.416431 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:23.416228    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" event={"ID":"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85","Type":"ContainerStarted","Data":"e6fa39f39cb7251caffb9fc3b8d098197c6d7683b1503960bc859b8c2d333887"}
Oct 30 19:42:23.641530 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:23.641471    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" podStartSLOduration=2.64145692 podStartE2EDuration="2.64145692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:23.591295022 +0000 UTC m=+1590.802512240" watchObservedRunningTime="2026-10-30 19:42:23.64145692 +0000 UTC m=+1590.852674063"
Oct 30 19:42:24.076214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:24.076141    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:24.078947 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:24.078608    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:42:24.078947 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:24.078662    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" podUID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:42:25.081841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.079244    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:42:25.081841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.079307    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" podUID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:42:25.900778 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.900742    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:25.916662 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.916635    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"]
Oct 30 19:42:25.921036 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.921017    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:25.921218 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.921202    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:25.998534 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.998504    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8tl\" (UniqueName: \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-kube-api-access-wf8tl\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:25.998709 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.998562    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:25.998709 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.998585    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:25.998709 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.998622    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:25.998709 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.998670    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:25.998709 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.998699    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:25.998907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.998877    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:25.998907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.998903    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:25.999060 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:25.999038    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.088839 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.088575    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:42:26.088839 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.088626    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" podUID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:42:26.100857 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.100827    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8tl\" (UniqueName: \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-kube-api-access-wf8tl\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.101019 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.100882    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.101019 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.100910    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.101019 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.100935    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.101019 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.100950    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.101019 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.100964    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.101327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.101036    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.101327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.101053    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.101327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.101102    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.101487 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.101454    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.101729 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.101708    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.102239 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.102217    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.102559 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.102539    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.102855 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.102833    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.105388 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.104638    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.106073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.106052    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.139343 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.139321    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"]
Oct 30 19:42:26.200020 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.193550    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8tl\" (UniqueName: \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-kube-api-access-wf8tl\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.200878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.200797    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.234982 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.234944    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:26.445793 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.445758    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:26.731006 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.730980    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"]
Oct 30 19:42:26.731520 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:42:26.731497    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4d6af5_bc40_4313_b009_2c1d61ee9eab.slice/crio-e7e07435bb4879c135e2910a0a11626335408c19e5f7249ae2a82128c9739e59 WatchSource:0}: Error finding container e7e07435bb4879c135e2910a0a11626335408c19e5f7249ae2a82128c9739e59: Status 404 returned error can't find the container with id e7e07435bb4879c135e2910a0a11626335408c19e5f7249ae2a82128c9739e59
Oct 30 19:42:26.734039 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.734001    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:26.734150 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.734080    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:26.734150 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:26.734125    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:27.070903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:27.070170    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:42:27.070903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:27.070225    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" podUID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:42:27.452210 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:27.452176    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" event={"ID":"7b4d6af5-bc40-4313-b009-2c1d61ee9eab","Type":"ContainerStarted","Data":"613c61d3c390c4458a616688595ae8843331d58b2ad024f12f0fae0caae05605"}
Oct 30 19:42:27.452633 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:27.452308    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" event={"ID":"7b4d6af5-bc40-4313-b009-2c1d61ee9eab","Type":"ContainerStarted","Data":"e7e07435bb4879c135e2910a0a11626335408c19e5f7249ae2a82128c9739e59"}
Oct 30 19:42:27.583855 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:27.582968    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" podStartSLOduration=2.582948603 podStartE2EDuration="2.582948603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:27.572241942 +0000 UTC m=+1594.783459106" watchObservedRunningTime="2026-10-30 19:42:27.582948603 +0000 UTC m=+1594.794165771"
Oct 30 19:42:28.080340 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:28.080301    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:42:28.080634 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:28.080607    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" podUID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:42:28.235776 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:28.235741    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:28.237398 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:28.237355    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:42:28.237590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:28.237567    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" podUID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:42:29.071840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:29.071224    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 19:42:29.071840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:29.071288    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" podUID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 19:42:29.235478 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:29.235445    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:42:29.235643 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:29.235498    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" podUID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:42:30.097396 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:30.097368    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:30.097909 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:30.097893    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:30.098734 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:30.098712    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 19:42:30.235487 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:30.235459    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:42:30.235659 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:30.235511    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" podUID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:42:31.235763 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:31.235728    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:42:31.236170 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:31.235778    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" podUID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:42:32.235503 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:32.235467    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:42:32.235680 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:32.235546    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" podUID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:42:33.235741 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:33.235706    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 19:42:33.236142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:33.235767    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" podUID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 19:42:34.497944 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:34.497913    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:34.498604 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:34.498580    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:34.499571 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:34.499550    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:42:35.198335 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:42:35.198298    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:42:37.799841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:37.798968    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:42:37.799841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:37.799746    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:42:37.824961 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:37.824918    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.399842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.395986    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:42:38.399842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.396898    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.399842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.397637    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.447840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.446791    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.482840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.477711    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.482840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.478142    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.824953 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.801916    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.824953 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.802285    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:42:38.824953 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.803006    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:42:38.824953 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.803066    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:42:38.850899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.827127    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:38.850899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:38.827540    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:42:39.544024 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.543958    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:39.576782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.576733    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:42:39.578359 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.578312    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:42:39.756843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.741160    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:42:39.799142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.799046    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:42:39.827841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.826922    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:42:39.861840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.757439    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:42:39.861840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.833399    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:42:39.861840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.833480    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:42:39.861840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.833537    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:42:39.861840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.833591    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:42:39.861840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.833664    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:40.033841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.990600    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:42:40.033841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:39.990673    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:40.033841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.023778    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:42:40.033841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.024339    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:42:40.033841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.028107    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:42:40.033841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.028599    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:42:40.033841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.028653    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:40.576555 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.576513    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:42:40.618840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.598556    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4" podUID="958d4afd-3115-469c-aa3f-ed96a8233238" containerName="main" probeResult="failure" output="Get \"https://10.133.0.30:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:40.618840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.601037    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:42:40.618840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.601331    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:42:40.786841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.782873    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:42:40.791976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.791933    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:40.852891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:40.835827    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 30 19:42:41.139902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:41.009848    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.139902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:41.019538    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.139902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:41.019580    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:42:41.139902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:41.109533    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.139902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:41.110482    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.139902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:41.113292    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:42:41.177838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:41.173895    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.535923 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:41.535831    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.582842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:41.579055    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.647644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:41.601502    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:41.647644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:41.601934    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:42:42.108841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.070196    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:42:42.123838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.072505    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:42:42.131839 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.097497    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:42:42.135897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.109909    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:42:42.136164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.111945    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:42.136543 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.117905    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:42:42.339687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.339653    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:42:42.373608 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.366613    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:42:42.457806 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.457776    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:42:42.474665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.474300    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:42:42.488783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.488763    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:42:42.491184 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.491165    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:42:42.495731 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.495711    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:42:42.496196 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.496179    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:42:42.522783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:42.522582    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:42:51.735622 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.735589    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"]
Oct 30 19:42:51.740169 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.740140    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.795732 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.795704    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.795732 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.795734    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.795899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.795859    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4dce860-24df-401c-a725-60bd751e2203-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.795942 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.795898    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.796012 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.795999    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4dce860-24df-401c-a725-60bd751e2203-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.796052 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.796041    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.796086 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.796057    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.796086 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.796071    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7rcf\" (UniqueName: \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-kube-api-access-s7rcf\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.796155 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.796094    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.897410 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.897382    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4dce860-24df-401c-a725-60bd751e2203-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.897590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.897437    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.897590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.897453    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.897590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.897471    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7rcf\" (UniqueName: \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-kube-api-access-s7rcf\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.897590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.897495    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.897804 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.897597    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.897804 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.897624    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.897804 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.897684    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4dce860-24df-401c-a725-60bd751e2203-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.897804 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.897733    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.898038 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.898009    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-credential-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.898098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.898078    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-socket\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.898361 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.898339    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-certs\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.898466 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.898391    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-data\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.898514 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.898485    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4dce860-24df-401c-a725-60bd751e2203-istiod-ca-cert\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.899872 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.899854    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-envoy\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.900236 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.900216    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4dce860-24df-401c-a725-60bd751e2203-istio-podinfo\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.949286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.949260    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"]
Oct 30 19:42:51.956855 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.956149    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7rcf\" (UniqueName: \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-kube-api-access-s7rcf\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:51.956855 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:51.956298    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-istio-token\") pod \"e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") " pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:52.054198 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:52.054101    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:52.500376 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:52.500350    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"]
Oct 30 19:42:52.500755 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:42:52.500730    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4dce860_24df_401c_a725_60bd751e2203.slice/crio-65cadd8354191cfb2ec3fcc894f7e91b5171b8a405e43027bda58b32b4e30e4a WatchSource:0}: Error finding container 65cadd8354191cfb2ec3fcc894f7e91b5171b8a405e43027bda58b32b4e30e4a: Status 404 returned error can't find the container with id 65cadd8354191cfb2ec3fcc894f7e91b5171b8a405e43027bda58b32b4e30e4a
Oct 30 19:42:52.503073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:52.503044    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:52.503149 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:52.503109    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:52.503149 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:52.503138    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:42:52.604758 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:52.604725    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" event={"ID":"d4dce860-24df-401c-a725-60bd751e2203","Type":"ContainerStarted","Data":"408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba"}
Oct 30 19:42:52.604969 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:52.604793    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" event={"ID":"d4dce860-24df-401c-a725-60bd751e2203","Type":"ContainerStarted","Data":"65cadd8354191cfb2ec3fcc894f7e91b5171b8a405e43027bda58b32b4e30e4a"}
Oct 30 19:42:52.733368 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:52.733299    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" podStartSLOduration=1.733281051 podStartE2EDuration="1.733281051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:52.72830711 +0000 UTC m=+1619.939524255" watchObservedRunningTime="2026-10-30 19:42:52.733281051 +0000 UTC m=+1619.944498198"
Oct 30 19:42:53.054600 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:53.054571    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:42:53.055780 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:53.055750    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:42:53.055984 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:53.055829    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:42:54.055329 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:54.055296    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:42:54.055908 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:54.055351    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:42:55.055378 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:55.055343    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:42:55.055783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:55.055400    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:42:56.055209 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:56.055173    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:42:56.055392 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:56.055250    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:42:57.054753 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:57.054721    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:42:57.055180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:57.054787    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:42:58.054478 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:58.054445    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:42:58.054650 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:58.054506    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:42:59.054926 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:59.054895    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 19:42:59.055471 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:42:59.054951    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 19:43:00.059146 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:00.059109    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:43:00.059604 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:00.059565    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:43:00.060194 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:00.060176    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:43:01.823246 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.823215    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"]
Oct 30 19:43:01.828724 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.828702    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.860484 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.860458    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.860610 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.860500    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhp5\" (UniqueName: \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-kube-api-access-vrhp5\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.860610 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.860521    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.860694 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.860645    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.860694 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.860680    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.860791 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.860779    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.860874 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.860852    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.860975 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.860955    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.861017 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.860989    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.922426 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.922392    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"]
Oct 30 19:43:01.962617 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.962584    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.962805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.962647    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhp5\" (UniqueName: \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-kube-api-access-vrhp5\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.962805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.962678    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.962805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.962718    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.962805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.962752    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.963074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.962828    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.963074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.962915    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.963074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.962995    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.963074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.963019    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.963074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.963056    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.963074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.963059    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-data\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.963391 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.963257    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-certs\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.963634 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.963605    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-credential-socket\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.963748 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.963655    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istiod-ca-cert\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.965369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.965344    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-envoy\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:01.965662 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:01.965643    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-podinfo\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:02.027754 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:02.027733    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-token\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:02.036915 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:02.036891    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhp5\" (UniqueName: \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-kube-api-access-vrhp5\") pod \"e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") " pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:02.142084 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:02.142067    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:02.570962 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:02.570929    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"]
Oct 30 19:43:02.571859 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:43:02.571833    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ccc82d_1202_459d_b4f1_fe6fa43d5abf.slice/crio-53b879be4247f4442fae453f04900961ca333cd72e9a0dd96e2f521c29b6088f WatchSource:0}: Error finding container 53b879be4247f4442fae453f04900961ca333cd72e9a0dd96e2f521c29b6088f: Status 404 returned error can't find the container with id 53b879be4247f4442fae453f04900961ca333cd72e9a0dd96e2f521c29b6088f
Oct 30 19:43:02.574308 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:02.574278    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:02.574374 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:02.574340    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:02.574374 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:02.574370    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:02.677269 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:02.677235    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" event={"ID":"18ccc82d-1202-459d-b4f1-fe6fa43d5abf","Type":"ContainerStarted","Data":"7f4b07275c95a464ab0c2a193497a0a3a912fe46c2a106491e03dbf8f79c5326"}
Oct 30 19:43:02.677417 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:02.677295    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" event={"ID":"18ccc82d-1202-459d-b4f1-fe6fa43d5abf","Type":"ContainerStarted","Data":"53b879be4247f4442fae453f04900961ca333cd72e9a0dd96e2f521c29b6088f"}
Oct 30 19:43:02.803400 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:02.803352    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" podStartSLOduration=1.803339754 podStartE2EDuration="1.803339754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:02.800742994 +0000 UTC m=+1630.011960140" watchObservedRunningTime="2026-10-30 19:43:02.803339754 +0000 UTC m=+1630.014556945"
Oct 30 19:43:03.142951 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:03.142903    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:03.144797 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:03.144778    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:43:03.144878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:03.144857    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 19:43:04.143438 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:04.143408    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:43:04.143837 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:04.143456    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 19:43:05.142760 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:05.142726    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:43:05.143036 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:05.142777    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 19:43:05.209593 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:43:05.209559    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:43:06.142883 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:06.142852    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:43:06.143056 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:06.142899    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 19:43:07.143023 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:07.142988    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:43:07.143431 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:07.143053    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 19:43:08.143241 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:08.143210    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:43:08.143669 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:08.143263    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 19:43:09.142972 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:09.142941    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 30 19:43:09.143141 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:09.142996    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 30 19:43:11.104180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:11.104148    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:11.106244 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:11.106227    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:11.108127 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:11.108109    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:43:12.091969 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:12.091930    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:43:12.092414 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:12.091894    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:12.092601 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:12.092401    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:12.093622 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:12.093599    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:43:12.094052 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:12.094031    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:43:12.124130 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:12.124091    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:43:12.572482 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:12.571327    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:35.198490 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:43:35.198455    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:43:42.029682 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:42.029641    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:43:42.035060 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:42.030265    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:42.095112 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:42.095065    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:42.127947 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:42.127901    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:42.547994 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:42.547940    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:42.603766 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:42.603724    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:42.604422 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:42.604044    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:43:42.873026 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:42.872918    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:43.037842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.036934    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:43.037842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.037368    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:43:43.105810 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.071162    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:43:43.119366 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.106088    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:43.156341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.151073    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:43:43.156341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.151491    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:43:43.156341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.152443    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:43:43.156341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.152761    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:43:43.156341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.153621    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:43:43.156341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.155404    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:43:43.579286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.578124    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:43.588836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.579885    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:43:43.619335 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.618928    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:43.693891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.619627    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:43.693891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.619895    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:43:43.693891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:43.619953    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:43:44.109588 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.038593    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.109588 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.077454    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:43:44.109588 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.089342    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:43:44.112348 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.112308    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.138097 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.138068    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:43:44.212842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.158031    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.392053 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.166987    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:43:44.392053 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.167024    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:43:44.395838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.393909    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:44.395838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.393957    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.415441 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.415407    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:43:44.415726 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.415696    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.600203 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.600139    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.763946 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.620166    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.795841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.643335    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:44.812848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:44.800985    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded"
Oct 30 19:43:45.094977 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.091217    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:45.322732 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.094905    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:43:45.326887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.168953    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:43:45.539992 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.539946    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:43:45.540702 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.540666    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:43:45.626927 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.626887    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:45.653517 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.636196    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:45.692251 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.692225    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:43:45.776866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.744333    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:43:45.776866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.759186    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:43:45.776866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.765389    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:43:45.787851 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.787132    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:43:45.805565 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.803441    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:43:45.818840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.816928    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:45.830224 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.830185    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:43:45.975969 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.975940    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:43:45.979886 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.979865    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:43:45.983833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:45.983795    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:43:46.020908 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:46.020875    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:43:47.161176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.161143    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"]
Oct 30 19:43:47.165462 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.165442    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.208648 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.208618    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-credential-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.208648 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.208651    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-data\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.208937 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.208741    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98483e4d-5bf0-4bae-a230-b0966a4c3512-istiod-ca-cert\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.208937 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.208841    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.208937 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.208897    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdg4d\" (UniqueName: \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-kube-api-access-tdg4d\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.209444 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.208965    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-certs\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.209444 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.209054    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-envoy\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.209444 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.209093    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-podinfo\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.209444 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.209130    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-token\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.226377 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.226344    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-94581ab2-openshift-default-dockercfg-l78tx\""
Oct 30 19:43:47.276327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.276297    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"]
Oct 30 19:43:47.310604 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.310566    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdg4d\" (UniqueName: \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-kube-api-access-tdg4d\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.310604 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.310606    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-certs\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.310878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.310699    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-envoy\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.310878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.310724    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-podinfo\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.310878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.310760    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-token\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.311089 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.310900    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-credential-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.311089 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.310928    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-data\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.311089 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.310988    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98483e4d-5bf0-4bae-a230-b0966a4c3512-istiod-ca-cert\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.311089 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.311059    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.311327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.311137    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-certs\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.311327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.311310    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-credential-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.311432 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.311349    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-socket\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.311486 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.311438    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-data\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.311805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.311786    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98483e4d-5bf0-4bae-a230-b0966a4c3512-istiod-ca-cert\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.313120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.313100    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-envoy\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.313429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.313412    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-podinfo\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.369334 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.369301    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-token\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.369502 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.369433    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdg4d\" (UniqueName: \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-kube-api-access-tdg4d\") pod \"e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") " pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.481894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.481794    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:47.942241 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:43:47.942201    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98483e4d_5bf0_4bae_a230_b0966a4c3512.slice/crio-a67c961cb2544735d22d46878c2c1128cef95742897cb02bb93ffab2760c4708 WatchSource:0}: Error finding container a67c961cb2544735d22d46878c2c1128cef95742897cb02bb93ffab2760c4708: Status 404 returned error can't find the container with id a67c961cb2544735d22d46878c2c1128cef95742897cb02bb93ffab2760c4708
Oct 30 19:43:47.945186 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.945146    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:47.945272 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.945244    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:47.945311 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.945288    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:43:47.949560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:47.949534    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"]
Oct 30 19:43:48.457311 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:48.457264    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" event={"ID":"98483e4d-5bf0-4bae-a230-b0966a4c3512","Type":"ContainerStarted","Data":"0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39"}
Oct 30 19:43:48.457759 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:48.457333    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" event={"ID":"98483e4d-5bf0-4bae-a230-b0966a4c3512","Type":"ContainerStarted","Data":"a67c961cb2544735d22d46878c2c1128cef95742897cb02bb93ffab2760c4708"}
Oct 30 19:43:48.482696 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:48.482674    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:48.681391 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:48.681328    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" podStartSLOduration=1.681314158 podStartE2EDuration="1.681314158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:48.630952579 +0000 UTC m=+1675.842169722" watchObservedRunningTime="2026-10-30 19:43:48.681314158 +0000 UTC m=+1675.892531302"
Oct 30 19:43:49.488605 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:49.488569    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:50.472760 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:50.472730    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:43:50.473706 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:43:50.473688    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:44:05.197296 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:44:05.197259    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:44:07.564717 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.564688    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"]
Oct 30 19:44:07.569123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.569106    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.626914 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.626875    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26p7r\" (UniqueName: \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-kube-api-access-26p7r\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.627091 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.626937    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.627155 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.627127    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.627252 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.627233    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.627366 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.627349    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.627517 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.627474    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.627517 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.627517    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.627680 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.627629    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0379827-bfec-4f2c-86b0-fb38be8d6281-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.627770 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.627751    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.682099 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.682062    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"]
Oct 30 19:44:07.728696 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.728668    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.728897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.728714    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26p7r\" (UniqueName: \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-kube-api-access-26p7r\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.728965 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.728900    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.729024 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.729007    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.729066 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.729053    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.729101 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.729086    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-data\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.729163 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.729145    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.729286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.729266    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.729412 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.729318    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.729412 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.729377    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-certs\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.729412 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.729381    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0379827-bfec-4f2c-86b0-fb38be8d6281-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.729582 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.729480    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-credential-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.729715 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.729695    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-socket\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.730270 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.730248    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0379827-bfec-4f2c-86b0-fb38be8d6281-istiod-ca-cert\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.731267 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.731245    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-envoy\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.731572 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.731553    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-podinfo\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.787591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.787560    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26p7r\" (UniqueName: \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-kube-api-access-26p7r\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.792390 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.792368    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-token\") pod \"e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") " pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:07.882418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:07.882331    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:08.325352 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:08.325321    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"]
Oct 30 19:44:08.325967 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:44:08.325946    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0379827_bfec_4f2c_86b0_fb38be8d6281.slice/crio-c24c80016f336fa08a8b47e45ef88310dd0164a6c2a3be23dc48f000fa23d09b WatchSource:0}: Error finding container c24c80016f336fa08a8b47e45ef88310dd0164a6c2a3be23dc48f000fa23d09b: Status 404 returned error can't find the container with id c24c80016f336fa08a8b47e45ef88310dd0164a6c2a3be23dc48f000fa23d09b
Oct 30 19:44:08.328481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:08.328448    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:08.328546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:08.328516    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:08.328580 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:08.328548    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:08.603618 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:08.603518    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" event={"ID":"d0379827-bfec-4f2c-86b0-fb38be8d6281","Type":"ContainerStarted","Data":"4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767"}
Oct 30 19:44:08.603618 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:08.603582    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" event={"ID":"d0379827-bfec-4f2c-86b0-fb38be8d6281","Type":"ContainerStarted","Data":"c24c80016f336fa08a8b47e45ef88310dd0164a6c2a3be23dc48f000fa23d09b"}
Oct 30 19:44:08.730471 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:08.730420    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" podStartSLOduration=1.7304051870000001 podStartE2EDuration="1.730405187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:08.726401319 +0000 UTC m=+1695.937618468" watchObservedRunningTime="2026-10-30 19:44:08.730405187 +0000 UTC m=+1695.941622331"
Oct 30 19:44:08.882986 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:08.882907    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:09.884703 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:09.884671    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:44:09.885192 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:09.884727    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:44:10.883282 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:10.883241    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:44:10.883492 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:10.883304    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:44:11.883405 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:11.883371    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:44:11.883777 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:11.883427    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:44:12.883303 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:12.883193    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:44:12.883303 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:12.883258    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:44:13.885222 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:13.885186    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:44:13.885856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:13.885800    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:44:14.389989 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:14.389940    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:14.394923 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:14.394887    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:14.604840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:14.603946    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:14.641018 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:14.640917    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:14.715853 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:14.714805    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:44:14.766841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:14.765894    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:14.803966 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:14.803914    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:14.898931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:14.891907    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:14.898931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:14.892998    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:44:14.898931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:14.893041    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:44:15.054461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:14.997620    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:15.054461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.053196    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:15.411931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.400419    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:44:15.411931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.401148    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:44:15.411931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.402163    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:15.411931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.402488    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:44:15.840989 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.840887    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:44:15.914840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.890221    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:44:15.917392 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.917355    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:44:15.918263 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.918238    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:44:15.973213 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.973168    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:44:15.995971 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.986635    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": context deadline exceeded"
Oct 30 19:44:15.996216 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:15.987777    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.026220 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.026179    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.051843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.033435    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:44:16.051843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.033495    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:44:16.051843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.033527    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:44:16.051843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.033565    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:44:16.051843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.033598    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:44:16.101897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.072464    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:44:16.221841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.207892    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:44:16.221841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.180911    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:44:16.221841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.103364    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:16.221841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.208267    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.221841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.208361    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:44:16.286952 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.283177    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:16.286952 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.283253    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.371948 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.346474    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:44:16.371948 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.346542    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.612605 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.610399    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.625949 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.622470    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded"
Oct 30 19:44:16.625949 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.622637    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:44:16.905841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.901192    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.905841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.901801    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 30 19:44:16.926839 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.906281    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:16.986914 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.957001    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:44:16.986914 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:16.957077    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:44:17.049844 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.033198    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:17.049844 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.033432    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:17.051860 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.050347    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:44:17.051860 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.050934    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:44:17.051860 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.051317    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:44:17.372459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.372355    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:17.626967 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.626892    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:17.662176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.660691    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:44:17.662462 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.662444    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:44:17.662697 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.662682    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:44:17.663572 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.663556    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:44:17.663735 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.663723    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:44:17.666198 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.666183    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:44:17.666355 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.666344    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:44:17.666466 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.666457    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:44:17.677960 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.677939    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:44:17.883498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.883414    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 19:44:17.883498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:17.883469    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 19:44:18.897323 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:18.897290    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:18.897688 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:18.897561    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:18.898431 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:18.898414    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:44:32.608580 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.608541    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"]
Oct 30 19:44:32.613516 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.613497    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.657228 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.657204    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"]
Oct 30 19:44:32.661211 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.661194    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.682535 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.682512    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-f2npg\""
Oct 30 19:44:32.722764 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.722730    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"]
Oct 30 19:44:32.762497 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.762467    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.762697 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.762519    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7sh8\" (UniqueName: \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-kube-api-access-j7sh8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.762697 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.762553    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.762697 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.762588    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.762697 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.762618    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.762999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.762768    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.762999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.762876    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8077dfc1-aa11-4d66-84db-932648ff157f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.762999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.762903    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.762999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.762923    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.763216 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.763012    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8077dfc1-aa11-4d66-84db-932648ff157f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.763216 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.763071    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.763216 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.763122    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.763216 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.763142    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.763428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.763223    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.763428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.763253    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ml6g\" (UniqueName: \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-kube-api-access-6ml6g\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.763428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.763371    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.763428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.763393    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.763428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.763418    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.771684 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.771658    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"]
Oct 30 19:44:32.777293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.777260    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.777722 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.777697    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"]
Oct 30 19:44:32.802559 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.802533    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"]
Oct 30 19:44:32.807296 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.807274    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:32.842920 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.842894    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"]
Oct 30 19:44:32.865465 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.865388    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.865726 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.865694    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.865860 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.865746    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.865860 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.865793    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.865860 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.865840    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.866044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.865893    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.866044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.865922    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.866044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.865967    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ml6g\" (UniqueName: \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-kube-api-access-6ml6g\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.866205 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866052    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.866205 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866115    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.866205 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866142    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.866205 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866193    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.866403 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866226    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.866403 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866273    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9h6w\" (UniqueName: \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-kube-api-access-b9h6w\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.866403 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866321    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7sh8\" (UniqueName: \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-kube-api-access-j7sh8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.866403 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866362    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.866403 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866383    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.866646 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866423    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.866646 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866469    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.866646 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866518    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.866646 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866564    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.866646 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866605    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8077dfc1-aa11-4d66-84db-932648ff157f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.866646 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866640    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.866958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866675    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.866958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866713    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.866958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866744    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.866958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.866798    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8077dfc1-aa11-4d66-84db-932648ff157f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.867633 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.867611    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8077dfc1-aa11-4d66-84db-932648ff157f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.867762 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.867738    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.867911 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.867808    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.868172 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.868146    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.868282 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.868258    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.868384 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.868360    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.868672 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.868648    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.868761 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.868744    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.869076 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.868941    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.869199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.869067    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.870878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.870859    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.871343 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.871301    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8077dfc1-aa11-4d66-84db-932648ff157f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.873056 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.873010    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.873315 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.873296    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.879725 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.877984    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"]
Oct 30 19:44:32.932530 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.932485    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7sh8\" (UniqueName: \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-kube-api-access-j7sh8\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.933361 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.933337    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ml6g\" (UniqueName: \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-kube-api-access-6ml6g\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.934179 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.934145    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-6mtks\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:32.935155 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.935133    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-vzbz4\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:32.968278 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968253    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:32.968399 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968298    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.968464 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968426    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.968464 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968455    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:32.968576 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968513    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:32.968644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968621    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9h6w\" (UniqueName: \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-kube-api-access-b9h6w\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.968740 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968724    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c49526a-3a13-4449-9658-b7a42840139e-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:32.968807 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968763    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.968807 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968793    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:32.968940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968831    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.968940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968921    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.969032 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968959    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c49526a-3a13-4449-9658-b7a42840139e-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:32.969032 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.968987    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.969032 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.969016    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.969181 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.969040    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.969181 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.969120    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:32.969283 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.969177    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.969283 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.969214    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.969283 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.969217    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.969283 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.969242    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncsh4\" (UniqueName: \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-kube-api-access-ncsh4\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:32.969470 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.969294    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:32.969508 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.969466    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.969738 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.969723    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.970860 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.970799    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.971315 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.971295    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:32.974479 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:32.974463    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:33.026443 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.026415    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:33.026641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.026624    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9h6w\" (UniqueName: \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-kube-api-access-b9h6w\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:33.071474 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.071398    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.071474 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.071438    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.071668 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.071507    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.071668 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.071545    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.071668 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.071616    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c49526a-3a13-4449-9658-b7a42840139e-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.071668 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.071640    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.071806 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.071687    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c49526a-3a13-4449-9658-b7a42840139e-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.071806 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.071709    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-credential-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.071942 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.071894    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-certs\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.071991 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.071954    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.072042 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.072012    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncsh4\" (UniqueName: \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-kube-api-access-ncsh4\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.072126 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.072104    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-data\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.072385 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.072357    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-socket\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.072523 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.072502    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c49526a-3a13-4449-9658-b7a42840139e-istiod-ca-cert\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.074310 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.074267    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-envoy\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.074460 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.074441    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c49526a-3a13-4449-9658-b7a42840139e-istio-podinfo\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.094689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.094662    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:33.144485 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.144453    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncsh4\" (UniqueName: \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-kube-api-access-ncsh4\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.145242 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.145221    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-istio-token\") pod \"e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") " pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.186678 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.186653    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"]
Oct 30 19:44:33.192164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.192146    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.229336 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.229304    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:33.247182 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.247159    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-94581ab2-serving-cert\""
Oct 30 19:44:33.258461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.258432    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"]
Oct 30 19:44:33.378752 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.378719    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6d9z\" (UniqueName: \"kubernetes.io/projected/9d911f78-17b3-426a-a9fe-ac57651bcb34-kube-api-access-r6d9z\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.378919 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.378891    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-metrics-certs\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.379019 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.379001    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d911f78-17b3-426a-a9fe-ac57651bcb34-openshift-service-ca\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.379213 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.379198    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-maas-api-tls\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.417808 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.417773    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"]
Oct 30 19:44:33.418297 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.418261    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.418461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.418440    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.448569 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.423698    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:33.453335 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.453286    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.484093 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.484057    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-maas-api-tls\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.484222 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.484168    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6d9z\" (UniqueName: \"kubernetes.io/projected/9d911f78-17b3-426a-a9fe-ac57651bcb34-kube-api-access-r6d9z\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.484277 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:44:33.484223    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-serving-cert: secret "maas-api-e2e-ait-94581ab2-serving-cert" not found
Oct 30 19:44:33.484315 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:44:33.484306    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-maas-api-tls podName:9d911f78-17b3-426a-a9fe-ac57651bcb34 nodeName:}" failed. No retries permitted until 2026-10-30 19:44:33.984283158 +0000 UTC m=+1721.195500289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-maas-api-tls") pod "maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf" (UID: "9d911f78-17b3-426a-a9fe-ac57651bcb34") : secret "maas-api-e2e-ait-94581ab2-serving-cert" not found
Oct 30 19:44:33.485878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.484319    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-metrics-certs\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.486214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.486190    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d911f78-17b3-426a-a9fe-ac57651bcb34-openshift-service-ca\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.488215 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.488193    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d911f78-17b3-426a-a9fe-ac57651bcb34-openshift-service-ca\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.493827 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:44:33.493786    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099d010e_c8ad_40cc_ac3b_f445bb1b52fa.slice/crio-3b33835b2524c38946de1b57a881c95107d16c6e2f1c55057b97a0ea920a30eb WatchSource:0}: Error finding container 3b33835b2524c38946de1b57a881c95107d16c6e2f1c55057b97a0ea920a30eb: Status 404 returned error can't find the container with id 3b33835b2524c38946de1b57a881c95107d16c6e2f1c55057b97a0ea920a30eb
Oct 30 19:44:33.493919 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.493896    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-metrics-certs\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.494460 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.494440    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"]
Oct 30 19:44:33.496943 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.496910    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.497031 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.496974    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.497031 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.497003    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.546116 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.545871    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6d9z\" (UniqueName: \"kubernetes.io/projected/9d911f78-17b3-426a-a9fe-ac57651bcb34-kube-api-access-r6d9z\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:33.645361 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.645326    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.645745 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.645394    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.645745 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.645422    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.645745 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.645579    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"]
Oct 30 19:44:33.755435 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.755396    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" event={"ID":"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6","Type":"ContainerStarted","Data":"a7e86214210b77384725cfc9dd04edd828eb2fcf29d296c7cae777fe2c675c88"}
Oct 30 19:44:33.759040 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.759002    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" event={"ID":"099d010e-c8ad-40cc-ac3b-f445bb1b52fa","Type":"ContainerStarted","Data":"8d81ffe9d7805dd09ff5b9d8904ab26a8c57d84cc021992d26316978c9e62140"}
Oct 30 19:44:33.759266 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.759245    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" event={"ID":"099d010e-c8ad-40cc-ac3b-f445bb1b52fa","Type":"ContainerStarted","Data":"3b33835b2524c38946de1b57a881c95107d16c6e2f1c55057b97a0ea920a30eb"}
Oct 30 19:44:33.762808 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.762789    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" event={"ID":"8077dfc1-aa11-4d66-84db-932648ff157f","Type":"ContainerStarted","Data":"cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885"}
Oct 30 19:44:33.763018 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.763001    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" event={"ID":"8077dfc1-aa11-4d66-84db-932648ff157f","Type":"ContainerStarted","Data":"29a389f56691d1455a5764339db781d72bc776dccfbcb8506846f79b3c830463"}
Oct 30 19:44:33.866529 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.865592    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"]
Oct 30 19:44:33.871108 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.871076    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.871208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.871144    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.871208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.871171    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:44:33.889006 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.887838    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" podStartSLOduration=1.887796202 podStartE2EDuration="1.887796202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:33.884396217 +0000 UTC m=+1721.095613362" watchObservedRunningTime="2026-10-30 19:44:33.887796202 +0000 UTC m=+1721.099013349"
Oct 30 19:44:33.975528 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:33.975504    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:34.004638 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.004608    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-maas-api-tls\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:34.015292 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.015228    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podStartSLOduration=2.015196207 podStartE2EDuration="2.015196207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:34.008312658 +0000 UTC m=+1721.219529828" watchObservedRunningTime="2026-10-30 19:44:34.015196207 +0000 UTC m=+1721.226413353"
Oct 30 19:44:34.020374 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.020343    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-maas-api-tls\") pod \"maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:34.095260 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.095230    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:34.096478 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.096449    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:44:34.096621 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.096524    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:44:34.108007 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.107987    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:34.520809 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.520761    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"]
Oct 30 19:44:34.527698 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.527672    4420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:44:34.827903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.824091    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" event={"ID":"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6","Type":"ContainerStarted","Data":"86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f"}
Oct 30 19:44:34.848292 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.848253    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf" event={"ID":"9d911f78-17b3-426a-a9fe-ac57651bcb34","Type":"ContainerStarted","Data":"d64f0e270848c1d716ca89289c735da9eaaa4f007c07c0cbc178c4bc6dfa65e3"}
Oct 30 19:44:34.851096 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.851065    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" event={"ID":"8c49526a-3a13-4449-9658-b7a42840139e","Type":"ContainerStarted","Data":"d0617c99a9206c5c539580b3a0870e597cb151dbdbaafdb59f92a1d6616c294c"}
Oct 30 19:44:34.851270 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:34.851121    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" event={"ID":"8c49526a-3a13-4449-9658-b7a42840139e","Type":"ContainerStarted","Data":"93e912ceb3fea537238d22ca021ec4681c60cb385ff2e74e2541160f47590804"}
Oct 30 19:44:35.007312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.007276    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6mtks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:44:35.007952 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.007918    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 19:44:35.016665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.016602    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" podStartSLOduration=3.016578652 podStartE2EDuration="3.016578652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:34.980262622 +0000 UTC m=+1722.191479802" watchObservedRunningTime="2026-10-30 19:44:35.016578652 +0000 UTC m=+1722.227795799"
Oct 30 19:44:35.095358 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.095258    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:44:35.095624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.095595    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:44:35.124711 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.124648    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" podStartSLOduration=3.124626826 podStartE2EDuration="3.124626826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:35.122517847 +0000 UTC m=+1722.333734996" watchObservedRunningTime="2026-10-30 19:44:35.124626826 +0000 UTC m=+1722.335843975"
Oct 30 19:44:35.252842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.245652    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:35.252842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.248206    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vzbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:44:35.252842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.248257    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:44:35.398915 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:44:35.390623    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:44:35.435741 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.433906    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:35.435741 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.435409    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:44:35.435741 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.435459    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:44:36.001899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.987795    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6mtks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:44:36.001899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:35.987870    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 19:44:36.072546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.072500    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:36.095243 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.095209    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:44:36.095705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.095676    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:44:36.259273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.240201    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vzbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:44:36.259273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.240263    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:44:36.396766 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.395934    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 30 19:44:36.437329 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.437204    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:44:36.437329 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.437270    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:44:36.594629 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.594524    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:36.597467 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.597432    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:44:36.598448 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.598423    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:44:36.604297 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.599907    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:36.638047 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.637999    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:36.688968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.688917    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:36.984782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:36.984733    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:37.001329 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.000289    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 30 19:44:37.002828 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.002787    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6mtks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:44:37.003500 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.003472    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 19:44:37.051279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.051232    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:37.084863 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.081458    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:37.084863 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.081889    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:44:37.115552 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.115514    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf" event={"ID":"9d911f78-17b3-426a-a9fe-ac57651bcb34","Type":"ContainerStarted","Data":"b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1"}
Oct 30 19:44:37.115803 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.115787    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:37.117249 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.117223    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:44:37.117440 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.117417    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:44:37.229705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.229673    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vzbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:44:37.230069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.230041    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:44:37.416911 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.404202    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:37.416911 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.404896    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:44:37.431110 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.431078    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:44:37.431380 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.431346    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:44:37.611839 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.600886    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 19:44:37.611839 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.601259    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 30 19:44:37.611839 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.601556    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:44:37.611839 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.601610    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:44:37.612361 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.612323    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:37.612878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.612497    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 30 19:44:37.617877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.613077    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:44:37.626005 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.625980    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:44:37.650101 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.650059    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:37.650721 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.650700    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:44:37.695998 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.695884    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf" podStartSLOduration=3.726219149 podStartE2EDuration="4.695865697s" totalImagesPullingTime="969.646548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:37.460776913 +0000 UTC m=+1724.671994058" watchObservedRunningTime="2026-10-30 19:44:37.695865697 +0000 UTC m=+1724.907082846"
Oct 30 19:44:37.725839 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.725778    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:37.873651 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.873617    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:44:37.976229 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.976136    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6mtks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:44:37.976461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:37.976433    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 19:44:38.058957 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.058910    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:38.064952 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.064921    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:44:38.264024 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.107368    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:38.326841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.243367    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vzbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:44:38.326841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.326328    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:44:38.326841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.243844    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:44:38.326841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.326396    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:44:38.404974 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.404924    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:38.472844 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.472677    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:44:38.472844 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.472736    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:44:38.617907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.609534    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:38.626938 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.614176    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:38.626938 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.614222    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:38.626938 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.615112    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:38.691842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.670134    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:38.878047 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:38.877930    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:44:39.003141 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.003104    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6mtks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.003448 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.003420    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 19:44:39.103251 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.103216    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.157043 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.157007    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:44:39.157307 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.113907    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:44:39.266872 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.246194    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vzbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.266872 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.246254    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:44:39.323843 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.321511    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:39.408902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.406909    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:39.430076 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.430023    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:44:39.433658 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.433625    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:44:39.433899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.433122    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 19:44:39.624974 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.622803    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:44:39.662266 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.625593    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:39.662266 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.650906    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:39.705850 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.705620    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 19:44:39.981664 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:39.972146    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 19:44:40.007946 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.004131    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6mtks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:44:40.015386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.008207    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 19:44:40.197158 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.197109    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:40.310961 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.237132    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:44:40.310961 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.237239    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:44:40.310961 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.255621    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vzbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:44:40.310961 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.255676    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:44:40.393943 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.392278    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 19:44:40.412760 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.412718    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:40.441901 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.441867    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:44:40.442167 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.442131    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:44:40.519046 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.518969    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:40.654997 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.654945    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:40.655579 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.655549    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:44:40.680838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.664917    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:40.680838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.665737    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:40.718457 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:40.716338    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.007340 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.007245    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6mtks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.007581 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.007548    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 19:44:41.007752 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.007733    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.095261 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.095225    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.095508 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.095480    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:44:41.200469 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.200422    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.204952 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.204879    4420 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Oct 30 19:44:41.252841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.233281    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vzbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.252841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.233332    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:44:41.394678 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.393909    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.430593 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.429183    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.430593 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.429980    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.430593 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.430016    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:44:41.656670 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.655963    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.670456 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.670415    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.722608 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.720455    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:44:41.741899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.741423    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 19:44:41.745379 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.745356    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 19:44:41.750289 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.749660    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 19:44:41.750289 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.750253    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 19:44:41.758125 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.758107    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 19:44:41.759177 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.759162    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 19:44:41.814189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.814147    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:44:41.980895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.977369    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6mtks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:44:41.980895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:41.977428    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 19:44:42.095580 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:42.095545    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 19:44:42.095754 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:42.095594    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 19:44:42.230075 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:42.229876    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vzbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:44:42.230075 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:42.229955    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:44:42.428265 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:42.428155    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:44:42.428265 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:42.428210    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:44:42.975346 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:42.975311    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6mtks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 19:44:42.975708 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:42.975664    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 19:44:43.230473 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:43.230373    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-vzbz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 30 19:44:43.230473 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:43.230432    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 30 19:44:43.285446 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:43.284965    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:44:43.431848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:43.430808    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 30 19:44:43.431848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:43.430884    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 30 19:44:43.608896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:43.608383    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:43.610294 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:43.609680    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:43.611209 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:43.611186    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:44:44.434168 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:44.434134    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:44.434902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:44.434880    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:44.435574 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:44.435556    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:44:44.718498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:44.718419    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:44.720927 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:44.720904    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:44.721579 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:44.721539    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 19:44:44.963901 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:44.963872    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"]
Oct 30 19:44:44.968178 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:44.968159    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.024648 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.024621    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d74c1bd9-serving-cert\""
Oct 30 19:44:45.053907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.053880    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"]
Oct 30 19:44:45.060962 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.060608    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-maas-api-tls\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.060962 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.060847    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7b859-0d3a-47ca-b023-f233695cf4d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.061600 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.061560    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8lj6\" (UniqueName: \"kubernetes.io/projected/cfd7b859-0d3a-47ca-b023-f233695cf4d0-kube-api-access-z8lj6\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.061778 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.061758    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-metrics-certs\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.123005 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.120650    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:45.123005 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.121903    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:45.123446 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.123424    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:44:45.162891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.162851    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8lj6\" (UniqueName: \"kubernetes.io/projected/cfd7b859-0d3a-47ca-b023-f233695cf4d0-kube-api-access-z8lj6\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.163059 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.163045    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-metrics-certs\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.163138 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.163122    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-maas-api-tls\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.163283 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.163262    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7b859-0d3a-47ca-b023-f233695cf4d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.163504 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:44:45.163296    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-serving-cert: secret "maas-api-e2e-migrate-d74c1bd9-serving-cert" not found
Oct 30 19:44:45.163504 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:44:45.163355    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-maas-api-tls podName:cfd7b859-0d3a-47ca-b023-f233695cf4d0 nodeName:}" failed. No retries permitted until 2026-10-30 19:44:45.66333617 +0000 UTC m=+1732.874553310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-maas-api-tls") pod "maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk" (UID: "cfd7b859-0d3a-47ca-b023-f233695cf4d0") : secret "maas-api-e2e-migrate-d74c1bd9-serving-cert" not found
Oct 30 19:44:45.164312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.164293    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7b859-0d3a-47ca-b023-f233695cf4d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.166558 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.166535    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-metrics-certs\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.229995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.229908    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8lj6\" (UniqueName: \"kubernetes.io/projected/cfd7b859-0d3a-47ca-b023-f233695cf4d0-kube-api-access-z8lj6\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.673418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.673388    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-maas-api-tls\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.676081 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.676056    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-maas-api-tls\") pod \"maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:45.882051 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:45.881971    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:46.319888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:46.313584    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"]
Oct 30 19:44:46.328247 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:44:46.328184    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd7b859_0d3a_47ca_b023_f233695cf4d0.slice/crio-38998281f5314ddfabcd91a1be2bb197cb84ca43da633f145d6672f85eb0d21d WatchSource:0}: Error finding container 38998281f5314ddfabcd91a1be2bb197cb84ca43da633f145d6672f85eb0d21d: Status 404 returned error can't find the container with id 38998281f5314ddfabcd91a1be2bb197cb84ca43da633f145d6672f85eb0d21d
Oct 30 19:44:46.699841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:46.699092    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk" event={"ID":"cfd7b859-0d3a-47ca-b023-f233695cf4d0","Type":"ContainerStarted","Data":"38998281f5314ddfabcd91a1be2bb197cb84ca43da633f145d6672f85eb0d21d"}
Oct 30 19:44:47.718549 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:47.713368    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk" event={"ID":"cfd7b859-0d3a-47ca-b023-f233695cf4d0","Type":"ContainerStarted","Data":"5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19"}
Oct 30 19:44:47.718549 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:47.713799    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:44:47.854838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:47.852291    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk" podStartSLOduration=2.956672799 podStartE2EDuration="3.852276365s" totalImagesPullingTime="895.603566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:47.846668031 +0000 UTC m=+1735.057885176" watchObservedRunningTime="2026-10-30 19:44:47.852276365 +0000 UTC m=+1735.063493510"
Oct 30 19:44:53.726525 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:44:53.726492    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:45:00.315871 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:00.315841    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"]
Oct 30 19:45:00.321568 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:00.321546    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"
Oct 30 19:45:00.384626 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:00.384592    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtrng\" (UniqueName: \"kubernetes.io/projected/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-kube-api-access-xtrng\") pod \"maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn\" (UID: \"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"
Oct 30 19:45:00.384795 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:00.384754    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn\" (UID: \"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"
Oct 30 19:45:00.413964 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:00.413935    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"]
Oct 30 19:45:00.442098 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:00.442057    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 19:45:00.486062 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:00.486034    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn\" (UID: \"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"
Oct 30 19:45:00.486308 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:00.486289    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtrng\" (UniqueName: \"kubernetes.io/projected/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-kube-api-access-xtrng\") pod \"maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn\" (UID: \"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"
Oct 30 19:45:00.486921 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:00.486895    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn\" (UID: \"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"
Oct 30 19:45:00.550185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:00.550143    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtrng\" (UniqueName: \"kubernetes.io/projected/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-kube-api-access-xtrng\") pod \"maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn\" (UID: \"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"
Oct 30 19:45:00.638069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:00.638048    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"
Oct 30 19:45:01.029443 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:01.029414    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"]
Oct 30 19:45:01.030336 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:45:01.030299    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bcafbb_36e2_4e1b_98b3_812d48ee1a4c.slice/crio-87e807fbbb7ca8be8c15544a0b72a0df3c01ed6594b0e6615c23f1278b1b3e78 WatchSource:0}: Error finding container 87e807fbbb7ca8be8c15544a0b72a0df3c01ed6594b0e6615c23f1278b1b3e78: Status 404 returned error can't find the container with id 87e807fbbb7ca8be8c15544a0b72a0df3c01ed6594b0e6615c23f1278b1b3e78
Oct 30 19:45:01.810144 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:01.810057    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn" event={"ID":"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c","Type":"ContainerStarted","Data":"1a7488b0c4af843b8dc22b60927a989e33fa7e56b8f0d9aecb9cf097a552a841"}
Oct 30 19:45:01.810144 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:01.810119    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn" event={"ID":"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c","Type":"ContainerStarted","Data":"87e807fbbb7ca8be8c15544a0b72a0df3c01ed6594b0e6615c23f1278b1b3e78"}
Oct 30 19:45:01.949915 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:01.949869    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn" podStartSLOduration=1.949855308 podStartE2EDuration="1.949855308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:01.947644703 +0000 UTC m=+1749.158861852" watchObservedRunningTime="2026-10-30 19:45:01.949855308 +0000 UTC m=+1749.161072452"
Oct 30 19:45:02.818319 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:02.818283    4420 generic.go:356] "Generic (PLEG): container finished" podID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerID="1a7488b0c4af843b8dc22b60927a989e33fa7e56b8f0d9aecb9cf097a552a841" exitCode=60
Oct 30 19:45:02.818729 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:02.818365    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn" event={"ID":"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c","Type":"ContainerDied","Data":"1a7488b0c4af843b8dc22b60927a989e33fa7e56b8f0d9aecb9cf097a552a841"}
Oct 30 19:45:02.818729 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:02.818593    4420 scope.go:122] "RemoveContainer" containerID="1a7488b0c4af843b8dc22b60927a989e33fa7e56b8f0d9aecb9cf097a552a841"
Oct 30 19:45:03.826598 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:03.826565    4420 generic.go:356] "Generic (PLEG): container finished" podID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerID="cfb4a1acc6e2db8ba603375dced14630e12e668cef1d9ddaabd31dcdfb8a7291" exitCode=60
Oct 30 19:45:03.827041 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:03.826631    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn" event={"ID":"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c","Type":"ContainerDied","Data":"cfb4a1acc6e2db8ba603375dced14630e12e668cef1d9ddaabd31dcdfb8a7291"}
Oct 30 19:45:03.827041 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:03.826672    4420 scope.go:122] "RemoveContainer" containerID="1a7488b0c4af843b8dc22b60927a989e33fa7e56b8f0d9aecb9cf097a552a841"
Oct 30 19:45:03.827041 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:03.826971    4420 scope.go:122] "RemoveContainer" containerID="cfb4a1acc6e2db8ba603375dced14630e12e668cef1d9ddaabd31dcdfb8a7291"
Oct 30 19:45:03.827381 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:45:03.827355    4420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn_odh-ai-gateway-infra(a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c"
Oct 30 19:45:04.835349 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:04.835276    4420 scope.go:122] "RemoveContainer" containerID="cfb4a1acc6e2db8ba603375dced14630e12e668cef1d9ddaabd31dcdfb8a7291"
Oct 30 19:45:04.836218 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:45:04.835569    4420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn_odh-ai-gateway-infra(a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c"
Oct 30 19:45:05.196093 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:45:05.196059    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:45:14.889880 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:14.889851    4420 scope.go:122] "RemoveContainer" containerID="cfb4a1acc6e2db8ba603375dced14630e12e668cef1d9ddaabd31dcdfb8a7291"
Oct 30 19:45:15.646908 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.639872    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"]
Oct 30 19:45:15.677840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.670094    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.724888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.723597    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-certs\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.724888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.723664    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-envoy\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.724888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.723764    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-data\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.724888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.723848    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-socket\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.724888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.723879    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-token\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.724888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.723912    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-podinfo\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.724888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.723944    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pflrq\" (UniqueName: \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-kube-api-access-pflrq\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.724888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.724064    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-credential-socket\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.724888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.724101    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istiod-ca-cert\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.734571 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.734544    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-2e5977-openshift-default-dockercfg-cjh8k\""
Oct 30 19:45:15.846876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.843039    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-certs\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.846876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.843118    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-envoy\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.846876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.843244    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-data\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.846876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.843320    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-socket\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.846876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.843350    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-token\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.846876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.843386    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-podinfo\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.846876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.843414    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pflrq\" (UniqueName: \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-kube-api-access-pflrq\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.846876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.843597    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-credential-socket\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.846876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.843637    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istiod-ca-cert\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.846876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.844521    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istiod-ca-cert\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.846876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.846352    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-certs\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.862646 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.858307    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"]
Oct 30 19:45:15.862646 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.859036    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-data\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.862646 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.859310    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-socket\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.899254 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.894923    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-credential-socket\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.907258 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.907181    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-envoy\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.912424 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.912390    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-podinfo\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.918207 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.918180    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-token\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.922896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.921884    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pflrq\" (UniqueName: \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-kube-api-access-pflrq\") pod \"e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") " pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:15.925764 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:15.925461    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn" event={"ID":"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c","Type":"ContainerStarted","Data":"e4a49634f8bdb0f24351ad675408d9e60a78b983ac42f9007af6d3784213296f"}
Oct 30 19:45:16.090044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:16.089994    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:16.611184 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:16.611153    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"]
Oct 30 19:45:16.611617 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:45:16.611596    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd22e64_3ad6_4f75_bc53_5dae3fccee49.slice/crio-1b62042a84463242f9f183d023c47534826732ab3cbcad02ad2e1747db9befa5 WatchSource:0}: Error finding container 1b62042a84463242f9f183d023c47534826732ab3cbcad02ad2e1747db9befa5: Status 404 returned error can't find the container with id 1b62042a84463242f9f183d023c47534826732ab3cbcad02ad2e1747db9befa5
Oct 30 19:45:16.614353 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:16.614323    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:16.614428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:16.614387    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:16.614428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:16.614414    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:16.933305 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:16.933271    4420 generic.go:356] "Generic (PLEG): container finished" podID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerID="e4a49634f8bdb0f24351ad675408d9e60a78b983ac42f9007af6d3784213296f" exitCode=60
Oct 30 19:45:16.933753 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:16.933333    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn" event={"ID":"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c","Type":"ContainerDied","Data":"e4a49634f8bdb0f24351ad675408d9e60a78b983ac42f9007af6d3784213296f"}
Oct 30 19:45:16.933753 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:16.933387    4420 scope.go:122] "RemoveContainer" containerID="cfb4a1acc6e2db8ba603375dced14630e12e668cef1d9ddaabd31dcdfb8a7291"
Oct 30 19:45:16.933918 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:16.933798    4420 scope.go:122] "RemoveContainer" containerID="e4a49634f8bdb0f24351ad675408d9e60a78b983ac42f9007af6d3784213296f"
Oct 30 19:45:16.934216 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:45:16.934193    4420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn_odh-ai-gateway-infra(a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c"
Oct 30 19:45:16.935604 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:16.935579    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" event={"ID":"9cd22e64-3ad6-4f75-bc53-5dae3fccee49","Type":"ContainerStarted","Data":"f1f7435d852f5be8e749468e4db038c213e716d7bdc4fd6d27a4c745b57c4af3"}
Oct 30 19:45:16.935708 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:16.935629    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" event={"ID":"9cd22e64-3ad6-4f75-bc53-5dae3fccee49","Type":"ContainerStarted","Data":"1b62042a84463242f9f183d023c47534826732ab3cbcad02ad2e1747db9befa5"}
Oct 30 19:45:17.087189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:17.086619    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"]
Oct 30 19:45:17.090362 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:17.090338    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:17.091966 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:17.091943    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:45:17.092080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:17.091986    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:45:18.076423 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.076355    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" podStartSLOduration=3.076335655 podStartE2EDuration="3.076335655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:17.185212681 +0000 UTC m=+1764.396429826" watchObservedRunningTime="2026-10-30 19:45:18.076335655 +0000 UTC m=+1765.287552800"
Oct 30 19:45:18.097295 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.097266    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:18.159087 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.159059    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"
Oct 30 19:45:18.211936 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.211876    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-kube-api-access-xtrng\" (UniqueName: \"kubernetes.io/projected/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-kube-api-access-xtrng\") pod \"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c\" (UID: \"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c\") "
Oct 30 19:45:18.212069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.211953    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-maas-api-service-ca\") pod \"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c\" (UID: \"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c\") "
Oct 30 19:45:18.212284 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.212251    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-maas-api-service-ca" pod "a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" (UID: "a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:45:18.212536 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.212521    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-maas-api-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:18.214026 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.214006    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-kube-api-access-xtrng" pod "a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" (UID: "a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c"). InnerVolumeSpecName "kube-api-access-xtrng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:18.313947 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.313918    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtrng\" (UniqueName: \"kubernetes.io/projected/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c-kube-api-access-xtrng\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:45:18.952879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.952845    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn" event={"ID":"a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c","Type":"ContainerDied","Data":"87e807fbbb7ca8be8c15544a0b72a0df3c01ed6594b0e6615c23f1278b1b3e78"}
Oct 30 19:45:18.953029 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.952918    4420 scope.go:122] "RemoveContainer" containerID="e4a49634f8bdb0f24351ad675408d9e60a78b983ac42f9007af6d3784213296f"
Oct 30 19:45:18.953029 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.952855    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"
Oct 30 19:45:18.953513 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.953489    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:18.954955 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:18.954935    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:45:19.250702 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:19.250609    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"]
Oct 30 19:45:19.255442 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:19.255416    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-94581ab2-29889825-hv7hn"]
Oct 30 19:45:20.896807 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:20.896767    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" path="/var/lib/kubelet/pods/a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c/volumes"
Oct 30 19:45:21.513976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:21.513944    4420 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:45:21.514138 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:21.514001    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:45:21.975927 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:21.975858    4420 generic.go:356] "Generic (PLEG): container finished" podID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerID="0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39" exitCode=0
Oct 30 19:45:21.976419 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:21.975943    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" event={"ID":"98483e4d-5bf0-4bae-a230-b0966a4c3512","Type":"ContainerDied","Data":"0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39"}
Oct 30 19:45:21.976419 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:21.976403    4420 scope.go:122] "RemoveContainer" containerID="0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39"
Oct 30 19:45:21.977831 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:21.977789    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:21.977898 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:21.977870    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:21.977940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:21.977904    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:45:22.992187 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:22.992142    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" event={"ID":"98483e4d-5bf0-4bae-a230-b0966a4c3512","Type":"ContainerStarted","Data":"dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e"}
Oct 30 19:45:23.482841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:23.482260    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:45:23.482841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:23.482396    4420 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:45:23.482841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:23.482436    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:45:24.483211 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:24.483173    4420 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:45:24.483600 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:24.483235    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:45:25.483270 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:25.483185    4420 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:45:25.483270 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:25.483232    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:45:26.483306 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:26.483263    4420 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:45:26.483862 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:26.483339    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:45:27.482214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:27.482177    4420 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 19:45:27.482486 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:27.482463    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 19:45:28.486664 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:28.486637    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:45:28.487124 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:28.487106    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:45:28.487697 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:28.487681    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:45:35.198711 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:45:35.198669    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:45:46.085047 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.085017    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"]
Oct 30 19:45:46.087049 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.085898    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerName="cleanup"
Oct 30 19:45:46.087049 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.085913    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerName="cleanup"
Oct 30 19:45:46.087049 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.085922    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerName="cleanup"
Oct 30 19:45:46.087049 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.085926    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerName="cleanup"
Oct 30 19:45:46.087049 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.086042    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerName="cleanup"
Oct 30 19:45:46.087049 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.086051    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerName="cleanup"
Oct 30 19:45:46.087049 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.086060    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerName="cleanup"
Oct 30 19:45:46.089589 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.089565    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.147981 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.147953    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-cf9b6f-serving-cert\""
Oct 30 19:45:46.198734 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.198704    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"]
Oct 30 19:45:46.228193 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.228163    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.228357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.228202    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69c92\" (UniqueName: \"kubernetes.io/projected/123635db-8c61-493e-b18d-e17afa4203bc-kube-api-access-69c92\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.228430 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.228403    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.228574 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.228551    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/123635db-8c61-493e-b18d-e17afa4203bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.329749 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.329716    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.329959 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.329758    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69c92\" (UniqueName: \"kubernetes.io/projected/123635db-8c61-493e-b18d-e17afa4203bc-kube-api-access-69c92\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.329959 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:45:46.329865    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-serving-cert: secret "maas-api-e2e-shared-a-w3-cf9b6f-serving-cert" not found
Oct 30 19:45:46.329959 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.329874    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.329959 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:45:46.329919    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-maas-api-tls podName:123635db-8c61-493e-b18d-e17afa4203bc nodeName:}" failed. No retries permitted until 2026-10-30 19:45:46.829903506 +0000 UTC m=+1794.041120633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-maas-api-tls") pod "maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd" (UID: "123635db-8c61-493e-b18d-e17afa4203bc") : secret "maas-api-e2e-shared-a-w3-cf9b6f-serving-cert" not found
Oct 30 19:45:46.330211 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.330031    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/123635db-8c61-493e-b18d-e17afa4203bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.330662 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.330637    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/123635db-8c61-493e-b18d-e17afa4203bc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.332446 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.332428    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.450640 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.450610    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69c92\" (UniqueName: \"kubernetes.io/projected/123635db-8c61-493e-b18d-e17afa4203bc-kube-api-access-69c92\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.841034 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.840941    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:46.843511 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:46.843481    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:47.006740 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:47.006702    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:47.432935 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:47.432908    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"]
Oct 30 19:45:47.433639 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:45:47.433601    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod123635db_8c61_493e_b18d_e17afa4203bc.slice/crio-55595b2aa79d3edafa291d7e153c2e8dc5d853d307598c565b03625561b66e55 WatchSource:0}: Error finding container 55595b2aa79d3edafa291d7e153c2e8dc5d853d307598c565b03625561b66e55: Status 404 returned error can't find the container with id 55595b2aa79d3edafa291d7e153c2e8dc5d853d307598c565b03625561b66e55
Oct 30 19:45:48.168802 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:48.168767    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd" event={"ID":"123635db-8c61-493e-b18d-e17afa4203bc","Type":"ContainerStarted","Data":"55595b2aa79d3edafa291d7e153c2e8dc5d853d307598c565b03625561b66e55"}
Oct 30 19:45:49.176752 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:49.176718    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd" event={"ID":"123635db-8c61-493e-b18d-e17afa4203bc","Type":"ContainerStarted","Data":"c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3"}
Oct 30 19:45:49.177179 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:49.176787    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:45:49.294238 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:49.294184    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd" podStartSLOduration=2.469349321 podStartE2EDuration="3.294170597s" totalImagesPullingTime="824.821276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:49.292226598 +0000 UTC m=+1796.503443743" watchObservedRunningTime="2026-10-30 19:45:49.294170597 +0000 UTC m=+1796.505387741"
Oct 30 19:45:55.190865 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:45:55.190798    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 19:46:05.206011 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:46:05.205962    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:46:14.495111 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.495079    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"]
Oct 30 19:46:14.495832 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.495805    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerName="cleanup"
Oct 30 19:46:14.495832 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.495833    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bcafbb-36e2-4e1b-98b3-812d48ee1a4c" containerName="cleanup"
Oct 30 19:46:14.500896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.500876    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.552535 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.552504    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-2e5977-serving-cert\""
Oct 30 19:46:14.605746 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.605712    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"]
Oct 30 19:46:14.659508 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.659475    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.659689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.659541    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.659689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.659667    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krbgr\" (UniqueName: \"kubernetes.io/projected/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-kube-api-access-krbgr\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.659854 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.659788    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.761208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.761126    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.761208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.761183    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.761406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.761259    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krbgr\" (UniqueName: \"kubernetes.io/projected/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-kube-api-access-krbgr\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.761406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.761331    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.762044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.762018    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.763725 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.763701    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.763861 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.763762    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:14.817400 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:14.817373    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krbgr\" (UniqueName: \"kubernetes.io/projected/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-kube-api-access-krbgr\") pod \"maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:15.115241 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:15.115155    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:15.494922 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:15.494894    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"]
Oct 30 19:46:15.496173 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:46:15.496149    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51a1ef9_c5e6_4a5e_ae69_edf73ea13fd7.slice/crio-19961a1d9483be0717b4e3f775de5e13766e74c4be674b70d595721defe955e1 WatchSource:0}: Error finding container 19961a1d9483be0717b4e3f775de5e13766e74c4be674b70d595721defe955e1: Status 404 returned error can't find the container with id 19961a1d9483be0717b4e3f775de5e13766e74c4be674b70d595721defe955e1
Oct 30 19:46:16.368076 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:16.368048    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth" event={"ID":"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7","Type":"ContainerStarted","Data":"19961a1d9483be0717b4e3f775de5e13766e74c4be674b70d595721defe955e1"}
Oct 30 19:46:17.376867 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:17.376804    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth" event={"ID":"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7","Type":"ContainerStarted","Data":"e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0"}
Oct 30 19:46:17.377327 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:17.376886    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:17.493747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:17.493691    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth" podStartSLOduration=2.608841242 podStartE2EDuration="3.493673767s" totalImagesPullingTime="884.832525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:17.490440868 +0000 UTC m=+1824.701658013" watchObservedRunningTime="2026-10-30 19:46:17.493673767 +0000 UTC m=+1824.704890930"
Oct 30 19:46:23.390461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.390429    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:46:23.558621 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.558588    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"]
Oct 30 19:46:23.558799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.558774    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf" podUID="9d911f78-17b3-426a-a9fe-ac57651bcb34" containerName="maas-api" containerID="cri-o://b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1" gracePeriod=30
Oct 30 19:46:23.884609 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.884585    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:46:23.901190 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.901165    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-metrics-certs\") pod \"9d911f78-17b3-426a-a9fe-ac57651bcb34\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") "
Oct 30 19:46:23.901350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.901272    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d911f78-17b3-426a-a9fe-ac57651bcb34-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d911f78-17b3-426a-a9fe-ac57651bcb34-openshift-service-ca\") pod \"9d911f78-17b3-426a-a9fe-ac57651bcb34\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") "
Oct 30 19:46:23.901489 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.901471    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d911f78-17b3-426a-a9fe-ac57651bcb34-kube-api-access-r6d9z\" (UniqueName: \"kubernetes.io/projected/9d911f78-17b3-426a-a9fe-ac57651bcb34-kube-api-access-r6d9z\") pod \"9d911f78-17b3-426a-a9fe-ac57651bcb34\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") "
Oct 30 19:46:23.901557 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.901505    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-maas-api-tls\") pod \"9d911f78-17b3-426a-a9fe-ac57651bcb34\" (UID: \"9d911f78-17b3-426a-a9fe-ac57651bcb34\") "
Oct 30 19:46:23.901703 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.901679    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d911f78-17b3-426a-a9fe-ac57651bcb34-openshift-service-ca" pod "9d911f78-17b3-426a-a9fe-ac57651bcb34" (UID: "9d911f78-17b3-426a-a9fe-ac57651bcb34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:23.902159 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.902129    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d911f78-17b3-426a-a9fe-ac57651bcb34-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:23.905324 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.904959    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-metrics-certs" pod "9d911f78-17b3-426a-a9fe-ac57651bcb34" (UID: "9d911f78-17b3-426a-a9fe-ac57651bcb34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:23.905324 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.905209    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d911f78-17b3-426a-a9fe-ac57651bcb34-kube-api-access-r6d9z" pod "9d911f78-17b3-426a-a9fe-ac57651bcb34" (UID: "9d911f78-17b3-426a-a9fe-ac57651bcb34"). InnerVolumeSpecName "kube-api-access-r6d9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:23.905324 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:23.905250    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-maas-api-tls" pod "9d911f78-17b3-426a-a9fe-ac57651bcb34" (UID: "9d911f78-17b3-426a-a9fe-ac57651bcb34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:24.006977 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.006901    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:24.006977 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.006930    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6d9z\" (UniqueName: \"kubernetes.io/projected/9d911f78-17b3-426a-a9fe-ac57651bcb34-kube-api-access-r6d9z\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:24.006977 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.006944    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d911f78-17b3-426a-a9fe-ac57651bcb34-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:24.425360 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.425324    4420 generic.go:356] "Generic (PLEG): container finished" podID="9d911f78-17b3-426a-a9fe-ac57651bcb34" containerID="b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1" exitCode=0
Oct 30 19:46:24.425857 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.425370    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"
Oct 30 19:46:24.425857 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.425410    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf" event={"ID":"9d911f78-17b3-426a-a9fe-ac57651bcb34","Type":"ContainerDied","Data":"b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1"}
Oct 30 19:46:24.425857 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.425470    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf" event={"ID":"9d911f78-17b3-426a-a9fe-ac57651bcb34","Type":"ContainerDied","Data":"d64f0e270848c1d716ca89289c735da9eaaa4f007c07c0cbc178c4bc6dfa65e3"}
Oct 30 19:46:24.425857 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.425502    4420 scope.go:122] "RemoveContainer" containerID="b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1"
Oct 30 19:46:24.436561 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.436545    4420 scope.go:122] "RemoveContainer" containerID="b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1"
Oct 30 19:46:24.436805 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:46:24.436787    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1\": container with ID starting with b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1 not found: ID does not exist" containerID="b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1"
Oct 30 19:46:24.436899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.436833    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1"} err="failed to get container status \"b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1\": rpc error: code = NotFound desc = could not find container \"b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1\": container with ID starting with b7ccadc8e963b1e1466597ef2e3e68f764ea5f6b0232c0fca2e2c763eee513d1 not found: ID does not exist"
Oct 30 19:46:24.600840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.600611    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"]
Oct 30 19:46:24.609624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.608008    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-94581ab2-589ff87c6b-jpqnf"]
Oct 30 19:46:24.896656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:24.896629    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d911f78-17b3-426a-a9fe-ac57651bcb34" path="/var/lib/kubelet/pods/9d911f78-17b3-426a-a9fe-ac57651bcb34/volumes"
Oct 30 19:46:25.801842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:25.801795    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"]
Oct 30 19:46:25.802292 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:25.802023    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy" containerID="cri-o://dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e" gracePeriod=30
Oct 30 19:46:28.489478 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:28.489438    4420 patch_prober.go:29] interesting pod/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:46:28.489898 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:28.489492    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:46:29.899689 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:29.899656    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"]
Oct 30 19:46:29.900156 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:29.899861    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk" podUID="cfd7b859-0d3a-47ca-b023-f233695cf4d0" containerName="maas-api" containerID="cri-o://5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19" gracePeriod=30
Oct 30 19:46:30.216128 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.215151    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:46:30.323112 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.323079    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd7b859-0d3a-47ca-b023-f233695cf4d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7b859-0d3a-47ca-b023-f233695cf4d0-openshift-service-ca\") pod \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") "
Oct 30 19:46:30.323308 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.323183    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-metrics-certs\") pod \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") "
Oct 30 19:46:30.323308 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.323250    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd7b859-0d3a-47ca-b023-f233695cf4d0-kube-api-access-z8lj6\" (UniqueName: \"kubernetes.io/projected/cfd7b859-0d3a-47ca-b023-f233695cf4d0-kube-api-access-z8lj6\") pod \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") "
Oct 30 19:46:30.323308 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.323301    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-maas-api-tls\") pod \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\" (UID: \"cfd7b859-0d3a-47ca-b023-f233695cf4d0\") "
Oct 30 19:46:30.323534 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.323507    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd7b859-0d3a-47ca-b023-f233695cf4d0-openshift-service-ca" pod "cfd7b859-0d3a-47ca-b023-f233695cf4d0" (UID: "cfd7b859-0d3a-47ca-b023-f233695cf4d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:30.323957 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.323931    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfd7b859-0d3a-47ca-b023-f233695cf4d0-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:30.325691 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.325662    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-maas-api-tls" pod "cfd7b859-0d3a-47ca-b023-f233695cf4d0" (UID: "cfd7b859-0d3a-47ca-b023-f233695cf4d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:30.325999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.325979    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd7b859-0d3a-47ca-b023-f233695cf4d0-kube-api-access-z8lj6" pod "cfd7b859-0d3a-47ca-b023-f233695cf4d0" (UID: "cfd7b859-0d3a-47ca-b023-f233695cf4d0"). InnerVolumeSpecName "kube-api-access-z8lj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:30.326067 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.326038    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-metrics-certs" pod "cfd7b859-0d3a-47ca-b023-f233695cf4d0" (UID: "cfd7b859-0d3a-47ca-b023-f233695cf4d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:46:30.425887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.425791    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:30.425887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.425840    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8lj6\" (UniqueName: \"kubernetes.io/projected/cfd7b859-0d3a-47ca-b023-f233695cf4d0-kube-api-access-z8lj6\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:30.425887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.425854    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfd7b859-0d3a-47ca-b023-f233695cf4d0-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:30.481317 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.477056    4420 generic.go:356] "Generic (PLEG): container finished" podID="cfd7b859-0d3a-47ca-b023-f233695cf4d0" containerID="5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19" exitCode=0
Oct 30 19:46:30.481317 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.477165    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk" event={"ID":"cfd7b859-0d3a-47ca-b023-f233695cf4d0","Type":"ContainerDied","Data":"5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19"}
Oct 30 19:46:30.481317 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.477216    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk" event={"ID":"cfd7b859-0d3a-47ca-b023-f233695cf4d0","Type":"ContainerDied","Data":"38998281f5314ddfabcd91a1be2bb197cb84ca43da633f145d6672f85eb0d21d"}
Oct 30 19:46:30.481317 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.477243    4420 scope.go:122] "RemoveContainer" containerID="5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19"
Oct 30 19:46:30.481317 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.477388    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"
Oct 30 19:46:30.493545 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.493526    4420 scope.go:122] "RemoveContainer" containerID="5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19"
Oct 30 19:46:30.493917 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:46:30.493890    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19\": container with ID starting with 5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19 not found: ID does not exist" containerID="5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19"
Oct 30 19:46:30.494035 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.493926    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19"} err="failed to get container status \"5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19\": rpc error: code = NotFound desc = could not find container \"5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19\": container with ID starting with 5e9632240feaaaecc323e04f3160419a68395fde99c80baad26f80b74aad4e19 not found: ID does not exist"
Oct 30 19:46:30.645014 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.644986    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"]
Oct 30 19:46:30.651924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.651896    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d74c1bd9-74d6f89697-wxstk"]
Oct 30 19:46:30.898343 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:30.898310    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd7b859-0d3a-47ca-b023-f233695cf4d0" path="/var/lib/kubelet/pods/cfd7b859-0d3a-47ca-b023-f233695cf4d0/volumes"
Oct 30 19:46:31.091032 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.091003    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:46:31.242925 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.242897    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-kube-api-access-tdg4d\" (UniqueName: \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-kube-api-access-tdg4d\") pod \"98483e4d-5bf0-4bae-a230-b0966a4c3512\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") "
Oct 30 19:46:31.243119 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.243019    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98483e4d-5bf0-4bae-a230-b0966a4c3512-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98483e4d-5bf0-4bae-a230-b0966a4c3512-istiod-ca-cert\") pod \"98483e4d-5bf0-4bae-a230-b0966a4c3512\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") "
Oct 30 19:46:31.243119 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.243071    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-socket\") pod \"98483e4d-5bf0-4bae-a230-b0966a4c3512\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") "
Oct 30 19:46:31.243119 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.243108    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-podinfo\") pod \"98483e4d-5bf0-4bae-a230-b0966a4c3512\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") "
Oct 30 19:46:31.243296 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.243221    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-credential-socket\") pod \"98483e4d-5bf0-4bae-a230-b0966a4c3512\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") "
Oct 30 19:46:31.243296 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.243273    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-data\") pod \"98483e4d-5bf0-4bae-a230-b0966a4c3512\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") "
Oct 30 19:46:31.243458 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.243438    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-envoy\") pod \"98483e4d-5bf0-4bae-a230-b0966a4c3512\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") "
Oct 30 19:46:31.243519 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.243479    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-token\" (UniqueName: \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-token\") pod \"98483e4d-5bf0-4bae-a230-b0966a4c3512\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") "
Oct 30 19:46:31.243578 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.243518    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-certs\") pod \"98483e4d-5bf0-4bae-a230-b0966a4c3512\" (UID: \"98483e4d-5bf0-4bae-a230-b0966a4c3512\") "
Oct 30 19:46:31.243631 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.243602    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-socket" pod "98483e4d-5bf0-4bae-a230-b0966a4c3512" (UID: "98483e4d-5bf0-4bae-a230-b0966a4c3512"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:31.243924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.243902    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98483e4d-5bf0-4bae-a230-b0966a4c3512-istiod-ca-cert" pod "98483e4d-5bf0-4bae-a230-b0966a4c3512" (UID: "98483e4d-5bf0-4bae-a230-b0966a4c3512"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:46:31.244132 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.244114    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-data" pod "98483e4d-5bf0-4bae-a230-b0966a4c3512" (UID: "98483e4d-5bf0-4bae-a230-b0966a4c3512"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:31.244370 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.244351    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98483e4d-5bf0-4bae-a230-b0966a4c3512-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:31.244455 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.244372    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:31.244455 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.244385    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:31.245536 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.245513    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-certs" pod "98483e4d-5bf0-4bae-a230-b0966a4c3512" (UID: "98483e4d-5bf0-4bae-a230-b0966a4c3512"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:31.245776 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.245750    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-credential-socket" pod "98483e4d-5bf0-4bae-a230-b0966a4c3512" (UID: "98483e4d-5bf0-4bae-a230-b0966a4c3512"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:31.246776 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.246752    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-envoy" pod "98483e4d-5bf0-4bae-a230-b0966a4c3512" (UID: "98483e4d-5bf0-4bae-a230-b0966a4c3512"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:46:31.247271 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.247244    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-kube-api-access-tdg4d" pod "98483e4d-5bf0-4bae-a230-b0966a4c3512" (UID: "98483e4d-5bf0-4bae-a230-b0966a4c3512"). InnerVolumeSpecName "kube-api-access-tdg4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:31.247688 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.247661    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-podinfo" pod "98483e4d-5bf0-4bae-a230-b0966a4c3512" (UID: "98483e4d-5bf0-4bae-a230-b0966a4c3512"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:46:31.247890 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.247874    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-token" pod "98483e4d-5bf0-4bae-a230-b0966a4c3512" (UID: "98483e4d-5bf0-4bae-a230-b0966a4c3512"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:46:31.345877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.345778    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:31.345877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.345806    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:31.345877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.345850    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:31.345877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.345863    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdg4d\" (UniqueName: \"kubernetes.io/projected/98483e4d-5bf0-4bae-a230-b0966a4c3512-kube-api-access-tdg4d\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:31.345877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.345876    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98483e4d-5bf0-4bae-a230-b0966a4c3512-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:31.346257 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.345889    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98483e4d-5bf0-4bae-a230-b0966a4c3512-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:46:31.486666 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.486635    4420 generic.go:356] "Generic (PLEG): container finished" podID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerID="dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e" exitCode=0
Oct 30 19:46:31.486879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.486683    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"
Oct 30 19:46:31.486879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.486708    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" event={"ID":"98483e4d-5bf0-4bae-a230-b0966a4c3512","Type":"ContainerDied","Data":"dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e"}
Oct 30 19:46:31.486879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.486767    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t" event={"ID":"98483e4d-5bf0-4bae-a230-b0966a4c3512","Type":"ContainerDied","Data":"a67c961cb2544735d22d46878c2c1128cef95742897cb02bb93ffab2760c4708"}
Oct 30 19:46:31.486879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.486798    4420 scope.go:122] "RemoveContainer" containerID="dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e"
Oct 30 19:46:31.498624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.498603    4420 scope.go:122] "RemoveContainer" containerID="0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39"
Oct 30 19:46:31.508360 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.508338    4420 scope.go:122] "RemoveContainer" containerID="dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e"
Oct 30 19:46:31.508653 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:46:31.508628    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e\": container with ID starting with dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e not found: ID does not exist" containerID="dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e"
Oct 30 19:46:31.508710 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.508662    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e"} err="failed to get container status \"dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e\": rpc error: code = NotFound desc = could not find container \"dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e\": container with ID starting with dbf35c4ae0d84c4927924cfb6836e94f974332480d1181cad323947371b7043e not found: ID does not exist"
Oct 30 19:46:31.508710 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.508680    4420 scope.go:122] "RemoveContainer" containerID="0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39"
Oct 30 19:46:31.509004 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:46:31.508985    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39\": container with ID starting with 0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39 not found: ID does not exist" containerID="0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39"
Oct 30 19:46:31.509056 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.509009    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39"} err="failed to get container status \"0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39\": rpc error: code = NotFound desc = could not find container \"0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39\": container with ID starting with 0384126d770b9801326621db205b29730755bbfe5924c0705db68f6a10241e39 not found: ID does not exist"
Oct 30 19:46:31.659996 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.659968    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"]
Oct 30 19:46:31.663593 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:31.663568    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-94581ab2-openshift-default-7d4b4fc485-pq82t"]
Oct 30 19:46:32.897828 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:32.897780    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" path="/var/lib/kubelet/pods/98483e4d-5bf0-4bae-a230-b0966a4c3512/volumes"
Oct 30 19:46:33.453195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.452207    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"]
Oct 30 19:46:33.454510 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.454486    4420 cpu_manager.go:414] "containerMap: removing container" podUID="cfd7b859-0d3a-47ca-b023-f233695cf4d0" containerName="maas-api"
Oct 30 19:46:33.464838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.454657    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd7b859-0d3a-47ca-b023-f233695cf4d0" containerName="maas-api"
Oct 30 19:46:33.465087 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.465066    4420 cpu_manager.go:414] "containerMap: removing container" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy"
Oct 30 19:46:33.465185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.465172    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy"
Oct 30 19:46:33.465292 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.465281    4420 cpu_manager.go:414] "containerMap: removing container" podUID="9d911f78-17b3-426a-a9fe-ac57651bcb34" containerName="maas-api"
Oct 30 19:46:33.465364 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.465355    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d911f78-17b3-426a-a9fe-ac57651bcb34" containerName="maas-api"
Oct 30 19:46:33.465440 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.465430    4420 cpu_manager.go:414] "containerMap: removing container" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy"
Oct 30 19:46:33.465505 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.465496    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy"
Oct 30 19:46:33.465856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.465839    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy"
Oct 30 19:46:33.465986 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.465976    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd7b859-0d3a-47ca-b023-f233695cf4d0" containerName="maas-api"
Oct 30 19:46:33.466069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.466061    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d911f78-17b3-426a-a9fe-ac57651bcb34" containerName="maas-api"
Oct 30 19:46:33.558209 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.558180    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"]
Oct 30 19:46:33.558545 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.558514    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.611784 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.611561    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-df789477-openshift-default-dockercfg-62x59\""
Oct 30 19:46:33.641146 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.641115    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-data\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.641335 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.641194    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-socket\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.641335 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.641230    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-credential-socket\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.641335 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.641319    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv9bl\" (UniqueName: \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-kube-api-access-bv9bl\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.641493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.641385    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istiod-ca-cert\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.641493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.641445    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-envoy\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.641577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.641549    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-certs\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.641634 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.641576    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-token\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.641634 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.641606    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-podinfo\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.742626 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.742539    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-certs\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.742626 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.742582    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-token\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.742626 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.742614    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-podinfo\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.742917 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.742688    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-data\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.742917 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.742717    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-socket\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.742917 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.742741    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-credential-socket\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.742917 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.742790    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv9bl\" (UniqueName: \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-kube-api-access-bv9bl\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.742917 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.742894    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istiod-ca-cert\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.743164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.742945    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-certs\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.743164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.742966    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-envoy\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.743164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.743068    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-data\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.743164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.743144    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-socket\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.743294 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.743192    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-credential-socket\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.743497 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.743474    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istiod-ca-cert\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.745281 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.745264    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-podinfo\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.745392 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.745321    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-envoy\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.801093 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.800619    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-token\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.801093 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.800623    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv9bl\" (UniqueName: \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-kube-api-access-bv9bl\") pod \"e2e-ait-df789477-openshift-default-5f45898cd9-wmssz\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") " pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:33.894252 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:33.894225    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:34.336747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:34.336672    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"]
Oct 30 19:46:34.373875 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:46:34.373843    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7829d2b9_e06d_41c6_98a7_01c17d9b29c8.slice/crio-fb805e5824a4e873e3929c1403cfec2dc7211d3a87c05b82fc70c3a5c71069ff WatchSource:0}: Error finding container fb805e5824a4e873e3929c1403cfec2dc7211d3a87c05b82fc70c3a5c71069ff: Status 404 returned error can't find the container with id fb805e5824a4e873e3929c1403cfec2dc7211d3a87c05b82fc70c3a5c71069ff
Oct 30 19:46:34.376548 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:34.376512    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:34.376654 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:34.376596    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:34.376654 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:34.376641    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:34.534916 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:34.533367    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" event={"ID":"7829d2b9-e06d-41c6-98a7-01c17d9b29c8","Type":"ContainerStarted","Data":"fb805e5824a4e873e3929c1403cfec2dc7211d3a87c05b82fc70c3a5c71069ff"}
Oct 30 19:46:35.212780 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:46:35.212735    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:46:35.541725 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:35.541642    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" event={"ID":"7829d2b9-e06d-41c6-98a7-01c17d9b29c8","Type":"ContainerStarted","Data":"099516948db982bb35c82430137ae1c3071fd897659898a3cf5bfdc48c1e3568"}
Oct 30 19:46:35.728848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:35.728771    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" podStartSLOduration=2.728757536 podStartE2EDuration="2.728757536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:35.723744106 +0000 UTC m=+1842.934961253" watchObservedRunningTime="2026-10-30 19:46:35.728757536 +0000 UTC m=+1842.939974681"
Oct 30 19:46:35.895415 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:35.895388    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:35.902173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:35.902142    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:36.549468 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:36.549439    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:36.550532 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:36.550510    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:46:48.841873 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:48.841065    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"]
Oct 30 19:46:48.842391 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:48.842215    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="98483e4d-5bf0-4bae-a230-b0966a4c3512" containerName="istio-proxy"
Oct 30 19:46:48.845641 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:48.845620    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:48.948286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:48.948181    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cf9b6f-openshift-default-dockercfg-qcqbf\""
Oct 30 19:46:48.953384 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:48.953174    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"]
Oct 30 19:46:48.975859 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:48.965990    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"]
Oct 30 19:46:48.975859 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:48.966107    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.036962 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.036928    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.037260 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.037235    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.037399 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.037383    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.037547 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.037528    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.037678 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.037661    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.037837 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.037799    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.038019 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.038000    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz92r\" (UniqueName: \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-kube-api-access-fz92r\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.038199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.038182    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ccb45-a25c-482a-9698-bada65a818bc-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.038334 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.038319    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d23027-83a7-4090-a3f6-2d22c3671881-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.038462 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.038446    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.038615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.038599    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.038732 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.038718    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.038875 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.038859    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.038996 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.038980    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.039163 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.039146    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.039314 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.039299    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.039439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.039424    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.039560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.039545    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhsf\" (UniqueName: \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-kube-api-access-fxhsf\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.048001 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.047803    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"]
Oct 30 19:46:49.140690 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.140649    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ccb45-a25c-482a-9698-bada65a818bc-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.140899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.140705    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d23027-83a7-4090-a3f6-2d22c3671881-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.140899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.140741    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.140899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.140788    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.140899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.140809    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.140899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.140850    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.140899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.140880    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.141437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.140940    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.141437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.140999    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.141437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.141032    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.141437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.141066    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhsf\" (UniqueName: \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-kube-api-access-fxhsf\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.141437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.141111    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.141437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.141176    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.141437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.141207    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.141437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.141262    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.141437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.141293    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.141437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.141340    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.141437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.141406    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz92r\" (UniqueName: \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-kube-api-access-fz92r\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.142018 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.141575    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ccb45-a25c-482a-9698-bada65a818bc-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.144848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.142278    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.144848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.142304    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.144848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.142534    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.144848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.142595    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.144848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.142863    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.144848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.143328    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-certs\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.144848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.143850    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-credential-socket\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.144848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.144600    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-data\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.151842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.150116    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.151842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.150254    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.153199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.153180    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d23027-83a7-4090-a3f6-2d22c3671881-istiod-ca-cert\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.153361 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.153180    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-envoy\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.153509 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.153469    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-podinfo\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.247719 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.247682    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhsf\" (UniqueName: \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-kube-api-access-fxhsf\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.247915 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.247742    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.248133 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.248108    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz92r\" (UniqueName: \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-kube-api-access-fz92r\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.254452 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.254402    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-token\") pod \"e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") " pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:49.432324 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.432227    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:49.490138 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:49.490103    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:50.101701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.101671    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"]
Oct 30 19:46:50.104490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.104451    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:50.104696 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.104672    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:50.104862 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.104840    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:50.397900 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:46:50.397291    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d23027_83a7_4090_a3f6_2d22c3671881.slice/crio-34e657256cb7cd115690171c490f3ab31c31d39c1f32e4939baedbe830ca985e WatchSource:0}: Error finding container 34e657256cb7cd115690171c490f3ab31c31d39c1f32e4939baedbe830ca985e: Status 404 returned error can't find the container with id 34e657256cb7cd115690171c490f3ab31c31d39c1f32e4939baedbe830ca985e
Oct 30 19:46:50.398203 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.398175    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"]
Oct 30 19:46:50.402868 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.402792    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:50.402986 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.402914    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:50.402986 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.402960    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:50.902120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.902082    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" event={"ID":"9c6ccb45-a25c-482a-9698-bada65a818bc","Type":"ContainerStarted","Data":"42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2"}
Oct 30 19:46:50.902305 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.902139    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" event={"ID":"9c6ccb45-a25c-482a-9698-bada65a818bc","Type":"ContainerStarted","Data":"5942cebb051ba9f7ef89e1674cf95813059e471a6d8e5391efb2fdaab835be3f"}
Oct 30 19:46:50.902305 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.902168    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" event={"ID":"f8d23027-83a7-4090-a3f6-2d22c3671881","Type":"ContainerStarted","Data":"033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1"}
Oct 30 19:46:50.902305 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:50.902191    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" event={"ID":"f8d23027-83a7-4090-a3f6-2d22c3671881","Type":"ContainerStarted","Data":"34e657256cb7cd115690171c490f3ab31c31d39c1f32e4939baedbe830ca985e"}
Oct 30 19:46:51.018032 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.017970    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" podStartSLOduration=3.017950526 podStartE2EDuration="3.017950526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:51.016278129 +0000 UTC m=+1858.227495297" watchObservedRunningTime="2026-10-30 19:46:51.017950526 +0000 UTC m=+1858.229167673"
Oct 30 19:46:51.140203 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.140155    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" podStartSLOduration=3.140138025 podStartE2EDuration="3.140138025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:51.135667335 +0000 UTC m=+1858.346884482" watchObservedRunningTime="2026-10-30 19:46:51.140138025 +0000 UTC m=+1858.351355170"
Oct 30 19:46:51.432969 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.432895    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:51.434946 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.434914    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:46:51.435102 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.434985    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" podUID="9c6ccb45-a25c-482a-9698-bada65a818bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:46:51.443753 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.443729    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"]
Oct 30 19:46:51.448373 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.448356    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.507842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.491769    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:51.507842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.502035    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 19:46:51.507842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.502116    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 19:46:51.507842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.502254    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2vd8g\""
Oct 30 19:46:51.507842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.502305    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:46:51.507842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.502324    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 19:46:51.507842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.502361    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:46:51.587632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.558385    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.587632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.558475    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.587632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.558503    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.587632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.558591    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d928ada-37ad-4ac0-baf0-ad3494327600-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.587632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.558710    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"]
Oct 30 19:46:51.587632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.558755    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqpqh\" (UniqueName: \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-kube-api-access-cqpqh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.587632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.558958    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.587632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.559053    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.714896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.708698    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.714896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.708749    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.714896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.708866    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d928ada-37ad-4ac0-baf0-ad3494327600-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.714896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.709038    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqpqh\" (UniqueName: \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-kube-api-access-cqpqh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.714896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.709112    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.714896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.709206    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.714896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.709299    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.723546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.721048    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d928ada-37ad-4ac0-baf0-ad3494327600-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.723546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.721065    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.723546 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.721708    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.723943 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.723922    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.724307 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.724290    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.806838 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.806405    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:51.818837 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:51.818600    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqpqh\" (UniqueName: \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-kube-api-access-cqpqh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kqkhh\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:52.096896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:52.092267    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:46:52.436196 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:52.436141    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:46:52.436680 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:52.436199    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" podUID="9c6ccb45-a25c-482a-9698-bada65a818bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:46:52.512507 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:52.512473    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:46:52.512707 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:52.512529    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:46:52.613262 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:52.613231    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"]
Oct 30 19:46:52.640543 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:46:52.640513    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d928ada_37ad_4ac0_baf0_ad3494327600.slice/crio-31bc581177f1987257516ce033055db9994a6b1f6c43f9f9480e23f919c14a2f WatchSource:0}: Error finding container 31bc581177f1987257516ce033055db9994a6b1f6c43f9f9480e23f919c14a2f: Status 404 returned error can't find the container with id 31bc581177f1987257516ce033055db9994a6b1f6c43f9f9480e23f919c14a2f
Oct 30 19:46:52.935288 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:52.935256    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh" event={"ID":"3d928ada-37ad-4ac0-baf0-ad3494327600","Type":"ContainerStarted","Data":"31bc581177f1987257516ce033055db9994a6b1f6c43f9f9480e23f919c14a2f"}
Oct 30 19:46:53.432782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:53.432748    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:46:53.432962 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:53.432806    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" podUID="9c6ccb45-a25c-482a-9698-bada65a818bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:46:53.499876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:53.499756    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:46:53.500451 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:53.500413    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:46:54.433697 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:54.433665    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:46:54.433895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:54.433714    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" podUID="9c6ccb45-a25c-482a-9698-bada65a818bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:46:54.507842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:54.503464    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:46:54.507842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:54.503520    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:46:55.432805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:55.432772    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:46:55.433074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:55.433051    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" podUID="9c6ccb45-a25c-482a-9698-bada65a818bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:46:55.490935 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:55.490886    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:46:55.491227 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:55.491193    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:46:56.433318 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:56.433268    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 30 19:46:56.433709 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:56.433348    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" podUID="9c6ccb45-a25c-482a-9698-bada65a818bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 30 19:46:56.491044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:56.491005    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:46:56.491194 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:56.491073    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:46:57.438677 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:57.438647    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:57.439230 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:57.439211    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:57.439941 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:57.439924    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:46:57.491633 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:57.491596    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 30 19:46:57.491810 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:57.491665    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 30 19:46:58.495312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:58.495283    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:58.495776 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:58.495759    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:58.496517 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:58.496497    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:46:59.004947 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:59.004916    4420 generic.go:356] "Generic (PLEG): container finished" podID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerID="f1f7435d852f5be8e749468e4db038c213e716d7bdc4fd6d27a4c745b57c4af3" exitCode=0
Oct 30 19:46:59.005149 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:59.004993    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" event={"ID":"9cd22e64-3ad6-4f75-bc53-5dae3fccee49","Type":"ContainerDied","Data":"f1f7435d852f5be8e749468e4db038c213e716d7bdc4fd6d27a4c745b57c4af3"}
Oct 30 19:46:59.005514 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:59.005497    4420 scope.go:122] "RemoveContainer" containerID="f1f7435d852f5be8e749468e4db038c213e716d7bdc4fd6d27a4c745b57c4af3"
Oct 30 19:46:59.007044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:59.007014    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:59.007144 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:59.007097    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:46:59.007206 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:46:59.007143    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:00.014889 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:00.014851    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" event={"ID":"9cd22e64-3ad6-4f75-bc53-5dae3fccee49","Type":"ContainerStarted","Data":"faf7011b49cd8743c04e227ef5e87e95b96464589ccc4c12bf947fbca6c3474a"}
Oct 30 19:47:00.090907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:00.090870    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:47:00.092154 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:00.092129    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:47:00.092280 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:00.092174    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:47:01.090946 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:01.090904    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:47:01.091347 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:01.090975    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:47:02.091141 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:02.091108    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:47:02.091687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:02.091657    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:47:03.091347 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.091312    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:47:03.091749 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.091367    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:47:03.845265 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.845225    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"]
Oct 30 19:47:03.849211 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.849192    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:03.914342 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.914308    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:03.914501 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.914381    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:03.914548 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.914523    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:03.914615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.914602    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:03.914666 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.914655    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:03.914710 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.914696    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:03.914753 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.914736    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:03.914845 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.914803    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:03.914888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.914874    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjx2j\" (UniqueName: \"kubernetes.io/projected/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-kube-api-access-zjx2j\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:03.953703 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:03.953677    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:47:04.016912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.016305    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.016912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.016435    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.016912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.016525    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.016912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.016593    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.016912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.016669    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.016912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.016700    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.016912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.016915    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.017388 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.016958    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjx2j\" (UniqueName: \"kubernetes.io/projected/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-kube-api-access-zjx2j\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.017388 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.017089    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.017388 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.017166    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.017872 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.017850    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.018876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.018854    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.019937 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.019916    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.020340 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.020314    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.020623 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.020603    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.022792 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.022769    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.056847 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.056796    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"]
Oct 30 19:47:04.089425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.089397    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjx2j\" (UniqueName: \"kubernetes.io/projected/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-kube-api-access-zjx2j\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.090370 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.090338    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:47:04.090506 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.090430    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:47:04.091202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.091175    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d386d8-73a7-4a53-8fec-2d70fcd9a2ef-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-cxf6d\" (UID: \"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.192541 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.192514    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:04.719021 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.716528    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"]
Oct 30 19:47:04.732424 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.729880    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:04.732424 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.729971    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:04.732424 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:04.730024    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:05.090885 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:05.090383    4420 patch_prober.go:29] interesting pod/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:47:05.090885 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:05.090441    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 30 19:47:05.099046 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:05.099007    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" event={"ID":"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef","Type":"ContainerStarted","Data":"a18352c2c9eb97e873e68a5ccbae7e2315f6c3edbd2a1e49c952227f1421d516"}
Oct 30 19:47:05.099202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:05.099075    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" event={"ID":"94d386d8-73a7-4a53-8fec-2d70fcd9a2ef","Type":"ContainerStarted","Data":"dd9c7f2cd0d89c7750dee4bd8975db343c93bf48035b82d1b8248e21d6bb279c"}
Oct 30 19:47:05.220503 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:05.219116    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:05.225337 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:05.224873    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podStartSLOduration=2.22485365 podStartE2EDuration="2.22485365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:05.2235592 +0000 UTC m=+1872.434776346" watchObservedRunningTime="2026-10-30 19:47:05.22485365 +0000 UTC m=+1872.436070794"
Oct 30 19:47:05.248902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:05.246117    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cxf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:47:05.248902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:05.246183    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podUID="94d386d8-73a7-4a53-8fec-2d70fcd9a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:47:05.266909 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:47:05.266871    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:47:06.097156 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:06.097127    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:47:06.098215 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:06.098195    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 19:47:06.193520 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:06.193490    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cxf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:47:06.193690 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:06.193547    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podUID="94d386d8-73a7-4a53-8fec-2d70fcd9a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:47:07.193325 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:07.193294    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cxf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:47:07.193797 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:07.193351    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podUID="94d386d8-73a7-4a53-8fec-2d70fcd9a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:47:07.824105 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:07.824070    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:07.824206 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:07.824137    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:47:08.126116 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:08.126016    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh" event={"ID":"3d928ada-37ad-4ac0-baf0-ad3494327600","Type":"ContainerStarted","Data":"a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3"}
Oct 30 19:47:08.126338 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:08.126306    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:47:08.128302 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:08.128274    4420 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kqkhh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:47:08.128440 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:08.128327    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh" podUID="3d928ada-37ad-4ac0-baf0-ad3494327600" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:47:08.193199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:08.193166    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cxf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:47:08.193384 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:08.193219    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podUID="94d386d8-73a7-4a53-8fec-2d70fcd9a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:47:08.343083 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:08.343030    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh" podStartSLOduration=2.163142181 podStartE2EDuration="17.343016123s" totalImagesPullingTime="15.179873942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:08.301117741 +0000 UTC m=+1875.512334882" watchObservedRunningTime="2026-10-30 19:47:08.343016123 +0000 UTC m=+1875.554233268"
Oct 30 19:47:09.134154 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:09.134114    4420 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kqkhh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:47:09.134309 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:09.134179    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh" podUID="3d928ada-37ad-4ac0-baf0-ad3494327600" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:47:09.193840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:09.193332    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cxf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:47:09.193840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:09.193440    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podUID="94d386d8-73a7-4a53-8fec-2d70fcd9a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:47:10.194113 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:10.194081    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cxf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:47:10.194524 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:10.194156    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podUID="94d386d8-73a7-4a53-8fec-2d70fcd9a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:47:11.193249 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:11.193220    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cxf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:47:11.193405 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:11.193277    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podUID="94d386d8-73a7-4a53-8fec-2d70fcd9a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:47:12.145452 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:12.145086    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:47:12.194260 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:12.194220    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cxf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:47:12.194551 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:12.194517    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podUID="94d386d8-73a7-4a53-8fec-2d70fcd9a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:47:13.193369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:13.193334    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cxf6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 19:47:13.193764 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:13.193393    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podUID="94d386d8-73a7-4a53-8fec-2d70fcd9a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 19:47:14.198191 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:14.198164    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:14.198665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:14.198650    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:14.199405 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:47:14.199390    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d"
Oct 30 19:47:35.202835 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:47:35.202776    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:48:05.200756 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:48:05.200664    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:48:10.220835 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:10.220780    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"]
Oct 30 19:48:10.221437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:10.220981    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" podUID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" containerName="istio-proxy" containerID="cri-o://7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75" gracePeriod=30
Oct 30 19:48:13.449664 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:13.449631    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:48:13.450097 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:13.449690    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" podUID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:48:15.503183 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.503127    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:48:15.594254 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594228    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-kube-api-access-jlz2f\" (UniqueName: \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-kube-api-access-jlz2f\") pod \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") "
Oct 30 19:48:15.594386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594266    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-socket\") pod \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") "
Oct 30 19:48:15.594386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594366    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istiod-ca-cert\") pod \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") "
Oct 30 19:48:15.594496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594432    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-envoy\") pod \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") "
Oct 30 19:48:15.594496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594476    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-credential-socket\") pod \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") "
Oct 30 19:48:15.594601 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594538    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-podinfo\") pod \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") "
Oct 30 19:48:15.594601 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594565    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-socket" pod "b5aab9b7-bc7c-44f4-8ec9-b335764484bc" (UID: "b5aab9b7-bc7c-44f4-8ec9-b335764484bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:15.594601 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594578    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-certs\") pod \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") "
Oct 30 19:48:15.594763 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594639    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-data\") pod \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") "
Oct 30 19:48:15.594763 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594727    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-token\" (UniqueName: \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-token\") pod \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\" (UID: \"b5aab9b7-bc7c-44f4-8ec9-b335764484bc\") "
Oct 30 19:48:15.594887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594779    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istiod-ca-cert" pod "b5aab9b7-bc7c-44f4-8ec9-b335764484bc" (UID: "b5aab9b7-bc7c-44f4-8ec9-b335764484bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:15.594940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594903    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-credential-socket" pod "b5aab9b7-bc7c-44f4-8ec9-b335764484bc" (UID: "b5aab9b7-bc7c-44f4-8ec9-b335764484bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:15.594940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594921    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-data" pod "b5aab9b7-bc7c-44f4-8ec9-b335764484bc" (UID: "b5aab9b7-bc7c-44f4-8ec9-b335764484bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:15.594940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.594929    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-certs" pod "b5aab9b7-bc7c-44f4-8ec9-b335764484bc" (UID: "b5aab9b7-bc7c-44f4-8ec9-b335764484bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:15.595479 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.595457    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:15.595606 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.595484    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:15.595606 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.595499    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:15.595606 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.595511    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:15.595606 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.595523    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:15.597064 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.597038    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-token" pod "b5aab9b7-bc7c-44f4-8ec9-b335764484bc" (UID: "b5aab9b7-bc7c-44f4-8ec9-b335764484bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:15.597064 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.597047    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-podinfo" pod "b5aab9b7-bc7c-44f4-8ec9-b335764484bc" (UID: "b5aab9b7-bc7c-44f4-8ec9-b335764484bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:15.597202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.597184    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-envoy" pod "b5aab9b7-bc7c-44f4-8ec9-b335764484bc" (UID: "b5aab9b7-bc7c-44f4-8ec9-b335764484bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:15.597245 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.597221    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-kube-api-access-jlz2f" pod "b5aab9b7-bc7c-44f4-8ec9-b335764484bc" (UID: "b5aab9b7-bc7c-44f4-8ec9-b335764484bc"). InnerVolumeSpecName "kube-api-access-jlz2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:15.697407 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.697378    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlz2f\" (UniqueName: \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-kube-api-access-jlz2f\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:15.697407 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.697407    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:15.697598 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.697418    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:15.697598 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.697427    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5aab9b7-bc7c-44f4-8ec9-b335764484bc-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:15.754722 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.754643    4420 generic.go:356] "Generic (PLEG): container finished" podID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" containerID="7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75" exitCode=0
Oct 30 19:48:15.754722 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.754705    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"
Oct 30 19:48:15.754958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.754725    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" event={"ID":"b5aab9b7-bc7c-44f4-8ec9-b335764484bc","Type":"ContainerDied","Data":"7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75"}
Oct 30 19:48:15.754958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.754810    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c" event={"ID":"b5aab9b7-bc7c-44f4-8ec9-b335764484bc","Type":"ContainerDied","Data":"3289a04ee8c5c65961fc7c1423e73249577935a925fa56a67c1d8ec305bf0ab5"}
Oct 30 19:48:15.754958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.754893    4420 scope.go:122] "RemoveContainer" containerID="7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75"
Oct 30 19:48:15.766171 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.766156    4420 scope.go:122] "RemoveContainer" containerID="7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75"
Oct 30 19:48:15.766460 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:48:15.766442    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75\": container with ID starting with 7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75 not found: ID does not exist" containerID="7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75"
Oct 30 19:48:15.766517 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.766469    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75"} err="failed to get container status \"7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75\": rpc error: code = NotFound desc = could not find container \"7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75\": container with ID starting with 7a2ca9681a026c560bbf8dbae9b8c9fd5986fcf2e871fb033e76430a4e876b75 not found: ID does not exist"
Oct 30 19:48:15.920576 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.920537    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"]
Oct 30 19:48:15.926319 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:15.926295    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-6hd5c"]
Oct 30 19:48:16.897064 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:16.897034    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" path="/var/lib/kubelet/pods/b5aab9b7-bc7c-44f4-8ec9-b335764484bc/volumes"
Oct 30 19:48:19.412229 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:19.412198    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"]
Oct 30 19:48:19.412657 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:19.412413    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy" containerID="cri-o://6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a" gracePeriod=30
Oct 30 19:48:19.461502 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:19.461472    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"]
Oct 30 19:48:19.461699 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:19.461668    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy" containerID="cri-o://030425a44b958edbe52de0ed98d45e513a783ce7e76ae2aaf8b91fdf22889a0e" gracePeriod=30
Oct 30 19:48:19.465286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:19.465264    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"]
Oct 30 19:48:19.465472 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:19.465438    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerName="istio-proxy" containerID="cri-o://7f4b07275c95a464ab0c2a193497a0a3a912fe46c2a106491e03dbf8f79c5326" gracePeriod=30
Oct 30 19:48:19.512407 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:19.512381    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"]
Oct 30 19:48:19.512587 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:19.512556    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy" containerID="cri-o://5abebb31940b25d226a8714f7afbc47095350eae061d6d4afa086452cef7ce4f" gracePeriod=30
Oct 30 19:48:23.779285 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:23.779252    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:48:23.779718 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:23.779312    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:48:23.794922 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:23.794891    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:48:23.795058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:23.794960    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:48:24.551210 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.551181    4420 patch_prober.go:29] interesting pod/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 19:48:24.551350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.551248    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 19:48:24.772500 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.772451    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:48:24.826616 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.826580    4420 generic.go:356] "Generic (PLEG): container finished" podID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerID="6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a" exitCode=0
Oct 30 19:48:24.826969 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.826788    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"
Oct 30 19:48:24.827048 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.827021    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" event={"ID":"a1ffc1f1-605c-4e82-ad64-cc1aee402b79","Type":"ContainerDied","Data":"6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a"}
Oct 30 19:48:24.827107 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.827081    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p" event={"ID":"a1ffc1f1-605c-4e82-ad64-cc1aee402b79","Type":"ContainerDied","Data":"5f92aa8d59bf5fc1499988357b397032aff06e51dc6ce92034645e5d73113bd6"}
Oct 30 19:48:24.827166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.827114    4420 scope.go:122] "RemoveContainer" containerID="6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a"
Oct 30 19:48:24.830063 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.830043    4420 generic.go:356] "Generic (PLEG): container finished" podID="36d86524-a126-407d-85ed-5a62b852046f" containerID="5abebb31940b25d226a8714f7afbc47095350eae061d6d4afa086452cef7ce4f" exitCode=0
Oct 30 19:48:24.830189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.830122    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" event={"ID":"36d86524-a126-407d-85ed-5a62b852046f","Type":"ContainerDied","Data":"5abebb31940b25d226a8714f7afbc47095350eae061d6d4afa086452cef7ce4f"}
Oct 30 19:48:24.832346 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.832321    4420 generic.go:356] "Generic (PLEG): container finished" podID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerID="030425a44b958edbe52de0ed98d45e513a783ce7e76ae2aaf8b91fdf22889a0e" exitCode=0
Oct 30 19:48:24.832461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.832412    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" event={"ID":"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53","Type":"ContainerDied","Data":"030425a44b958edbe52de0ed98d45e513a783ce7e76ae2aaf8b91fdf22889a0e"}
Oct 30 19:48:24.834942 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.834276    4420 generic.go:356] "Generic (PLEG): container finished" podID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerID="7f4b07275c95a464ab0c2a193497a0a3a912fe46c2a106491e03dbf8f79c5326" exitCode=0
Oct 30 19:48:24.834942 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.834315    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" event={"ID":"18ccc82d-1202-459d-b4f1-fe6fa43d5abf","Type":"ContainerDied","Data":"7f4b07275c95a464ab0c2a193497a0a3a912fe46c2a106491e03dbf8f79c5326"}
Oct 30 19:48:24.840185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.838993    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-token\" (UniqueName: \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-token\") pod \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") "
Oct 30 19:48:24.840185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.839084    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-data\") pod \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") "
Oct 30 19:48:24.840185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.839141    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-envoy\") pod \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") "
Oct 30 19:48:24.840185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.839189    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-socket\") pod \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") "
Oct 30 19:48:24.840185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.839241    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-kube-api-access-8wbzd\" (UniqueName: \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-kube-api-access-8wbzd\") pod \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") "
Oct 30 19:48:24.840185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.839302    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istiod-ca-cert\") pod \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") "
Oct 30 19:48:24.840185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.839343    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-podinfo\") pod \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") "
Oct 30 19:48:24.840185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.839486    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-credential-socket\") pod \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") "
Oct 30 19:48:24.840185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.839554    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-certs\") pod \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\" (UID: \"a1ffc1f1-605c-4e82-ad64-cc1aee402b79\") "
Oct 30 19:48:24.840781 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.840659    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-certs" pod "a1ffc1f1-605c-4e82-ad64-cc1aee402b79" (UID: "a1ffc1f1-605c-4e82-ad64-cc1aee402b79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:24.841692 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.841603    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-socket" pod "a1ffc1f1-605c-4e82-ad64-cc1aee402b79" (UID: "a1ffc1f1-605c-4e82-ad64-cc1aee402b79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:24.841692 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.841665    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-data" pod "a1ffc1f1-605c-4e82-ad64-cc1aee402b79" (UID: "a1ffc1f1-605c-4e82-ad64-cc1aee402b79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:24.841906 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.841755    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-credential-socket" pod "a1ffc1f1-605c-4e82-ad64-cc1aee402b79" (UID: "a1ffc1f1-605c-4e82-ad64-cc1aee402b79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:24.841970 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.841911    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istiod-ca-cert" pod "a1ffc1f1-605c-4e82-ad64-cc1aee402b79" (UID: "a1ffc1f1-605c-4e82-ad64-cc1aee402b79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:24.845248 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.845221    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-envoy" pod "a1ffc1f1-605c-4e82-ad64-cc1aee402b79" (UID: "a1ffc1f1-605c-4e82-ad64-cc1aee402b79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:24.845507 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.845482    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-token" pod "a1ffc1f1-605c-4e82-ad64-cc1aee402b79" (UID: "a1ffc1f1-605c-4e82-ad64-cc1aee402b79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:24.845717 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.845697    4420 scope.go:122] "RemoveContainer" containerID="6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a"
Oct 30 19:48:24.846234 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:48:24.846059    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a\": container with ID starting with 6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a not found: ID does not exist" containerID="6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a"
Oct 30 19:48:24.846234 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.846095    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a"} err="failed to get container status \"6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a\": rpc error: code = NotFound desc = could not find container \"6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a\": container with ID starting with 6ed88461e3b7daf1b4ab0cd34422f49a24ef4fcea49d75fe6442f3900e7bb42a not found: ID does not exist"
Oct 30 19:48:24.846234 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.846121    4420 scope.go:122] "RemoveContainer" containerID="59ece95686bb3dc10cbdd236182b2afdb39a93b02c77570e42cf4c8d4db7711f"
Oct 30 19:48:24.847481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.847454    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-kube-api-access-8wbzd" pod "a1ffc1f1-605c-4e82-ad64-cc1aee402b79" (UID: "a1ffc1f1-605c-4e82-ad64-cc1aee402b79"). InnerVolumeSpecName "kube-api-access-8wbzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:24.851775 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.851746    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-podinfo" pod "a1ffc1f1-605c-4e82-ad64-cc1aee402b79" (UID: "a1ffc1f1-605c-4e82-ad64-cc1aee402b79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:24.870176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.870151    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:48:24.907226 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.907204    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:48:24.910749 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.910729    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:48:24.941133 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941105    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-credential-socket\") pod \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") "
Oct 30 19:48:24.941273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941259    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-socket\") pod \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") "
Oct 30 19:48:24.941339 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941290    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-credential-socket" pod "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" (UID: "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:24.941339 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941305    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-token\" (UniqueName: \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-token\") pod \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") "
Oct 30 19:48:24.941443 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941422    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istiod-ca-cert\") pod \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") "
Oct 30 19:48:24.941551 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941509    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-kube-api-access-q8zlx\" (UniqueName: \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-kube-api-access-q8zlx\") pod \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") "
Oct 30 19:48:24.941673 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941570    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-socket" pod "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" (UID: "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:24.941673 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941572    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-podinfo\") pod \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") "
Oct 30 19:48:24.941774 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941684    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-data\") pod \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") "
Oct 30 19:48:24.941858 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941781    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istiod-ca-cert" pod "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" (UID: "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:24.941858 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941810    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-envoy\") pod \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") "
Oct 30 19:48:24.941967 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.941927    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-certs\") pod \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\" (UID: \"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53\") "
Oct 30 19:48:24.942210 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.942171    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-certs" pod "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" (UID: "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:24.942393 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.942367    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-data" pod "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" (UID: "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:24.944156 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944101    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944156 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944130    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944156 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944145    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944160    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944177    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944190    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944204    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbzd\" (UniqueName: \"kubernetes.io/projected/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-kube-api-access-8wbzd\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944212    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944221    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944230    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944238    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944251    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944294    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1ffc1f1-605c-4e82-ad64-cc1aee402b79-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.944354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944305    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:24.945033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944705    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-podinfo" pod "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" (UID: "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:24.945033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944762    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-envoy" pod "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" (UID: "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:24.945033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.944787    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-token" pod "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" (UID: "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:24.945777 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:24.945758    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-kube-api-access-q8zlx" pod "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" (UID: "570a77ab-2b41-4cb5-8b6d-5231cf2f7c53"). InnerVolumeSpecName "kube-api-access-q8zlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:25.044845 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.044751    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d86524-a126-407d-85ed-5a62b852046f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d86524-a126-407d-85ed-5a62b852046f-istiod-ca-cert\") pod \"36d86524-a126-407d-85ed-5a62b852046f\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") "
Oct 30 19:48:25.044845 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.044791    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-kube-api-access-vrhp5\" (UniqueName: \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-kube-api-access-vrhp5\") pod \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") "
Oct 30 19:48:25.045026 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.044846    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-certs\") pod \"36d86524-a126-407d-85ed-5a62b852046f\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") "
Oct 30 19:48:25.045026 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.044924    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36d86524-a126-407d-85ed-5a62b852046f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d86524-a126-407d-85ed-5a62b852046f-istio-podinfo\") pod \"36d86524-a126-407d-85ed-5a62b852046f\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") "
Oct 30 19:48:25.045026 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.044963    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-token\" (UniqueName: \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-token\") pod \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") "
Oct 30 19:48:25.045174 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045026    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-data\") pod \"36d86524-a126-407d-85ed-5a62b852046f\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") "
Oct 30 19:48:25.045174 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045055    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istiod-ca-cert\") pod \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") "
Oct 30 19:48:25.045174 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045152    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-credential-socket\") pod \"36d86524-a126-407d-85ed-5a62b852046f\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") "
Oct 30 19:48:25.045323 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045170    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d86524-a126-407d-85ed-5a62b852046f-istiod-ca-cert" pod "36d86524-a126-407d-85ed-5a62b852046f" (UID: "36d86524-a126-407d-85ed-5a62b852046f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:25.045323 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045221    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-istio-token\" (UniqueName: \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-istio-token\") pod \"36d86524-a126-407d-85ed-5a62b852046f\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") "
Oct 30 19:48:25.045323 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045222    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-certs" pod "36d86524-a126-407d-85ed-5a62b852046f" (UID: "36d86524-a126-407d-85ed-5a62b852046f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.045323 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045261    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-data\") pod \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") "
Oct 30 19:48:25.045323 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045290    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-certs\") pod \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") "
Oct 30 19:48:25.045573 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045328    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-kube-api-access-x5h8d\" (UniqueName: \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-kube-api-access-x5h8d\") pod \"36d86524-a126-407d-85ed-5a62b852046f\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") "
Oct 30 19:48:25.045573 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045363    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-socket\") pod \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") "
Oct 30 19:48:25.045573 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045429    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-podinfo\") pod \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") "
Oct 30 19:48:25.045573 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045449    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-credential-socket" pod "36d86524-a126-407d-85ed-5a62b852046f" (UID: "36d86524-a126-407d-85ed-5a62b852046f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.045573 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045466    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-socket\") pod \"36d86524-a126-407d-85ed-5a62b852046f\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") "
Oct 30 19:48:25.045573 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045552    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-envoy\") pod \"36d86524-a126-407d-85ed-5a62b852046f\" (UID: \"36d86524-a126-407d-85ed-5a62b852046f\") "
Oct 30 19:48:25.045893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045582    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istiod-ca-cert" pod "18ccc82d-1202-459d-b4f1-fe6fa43d5abf" (UID: "18ccc82d-1202-459d-b4f1-fe6fa43d5abf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:48:25.045893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045635    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-data" pod "36d86524-a126-407d-85ed-5a62b852046f" (UID: "36d86524-a126-407d-85ed-5a62b852046f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.045893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045647    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-credential-socket\") pod \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") "
Oct 30 19:48:25.045893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045717    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-envoy\") pod \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\" (UID: \"18ccc82d-1202-459d-b4f1-fe6fa43d5abf\") "
Oct 30 19:48:25.045893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045849    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-data" pod "18ccc82d-1202-459d-b4f1-fe6fa43d5abf" (UID: "18ccc82d-1202-459d-b4f1-fe6fa43d5abf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.045893 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.045879    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-certs" pod "18ccc82d-1202-459d-b4f1-fe6fa43d5abf" (UID: "18ccc82d-1202-459d-b4f1-fe6fa43d5abf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.046200 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.046010    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-socket" pod "36d86524-a126-407d-85ed-5a62b852046f" (UID: "36d86524-a126-407d-85ed-5a62b852046f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.046200 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.046075    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-socket" pod "18ccc82d-1202-459d-b4f1-fe6fa43d5abf" (UID: "18ccc82d-1202-459d-b4f1-fe6fa43d5abf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.046833 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.046761    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.046968    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8zlx\" (UniqueName: \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-kube-api-access-q8zlx\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.046963    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-credential-socket" pod "18ccc82d-1202-459d-b4f1-fe6fa43d5abf" (UID: "18ccc82d-1202-459d-b4f1-fe6fa43d5abf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.046991    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.047006    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.047020    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.047033    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.047046    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.047058    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.047071    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.047084    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.047098    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.047111    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.047142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.047124    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36d86524-a126-407d-85ed-5a62b852046f-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.048104 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.047940    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36d86524-a126-407d-85ed-5a62b852046f-istio-podinfo" pod "36d86524-a126-407d-85ed-5a62b852046f" (UID: "36d86524-a126-407d-85ed-5a62b852046f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:25.048369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.048342    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-kube-api-access-vrhp5" pod "18ccc82d-1202-459d-b4f1-fe6fa43d5abf" (UID: "18ccc82d-1202-459d-b4f1-fe6fa43d5abf"). InnerVolumeSpecName "kube-api-access-vrhp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:25.048369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.048378    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-token" pod "18ccc82d-1202-459d-b4f1-fe6fa43d5abf" (UID: "18ccc82d-1202-459d-b4f1-fe6fa43d5abf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:25.048655 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.048636    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-kube-api-access-x5h8d" pod "36d86524-a126-407d-85ed-5a62b852046f" (UID: "36d86524-a126-407d-85ed-5a62b852046f"). InnerVolumeSpecName "kube-api-access-x5h8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:25.048736 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.048719    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-podinfo" pod "18ccc82d-1202-459d-b4f1-fe6fa43d5abf" (UID: "18ccc82d-1202-459d-b4f1-fe6fa43d5abf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:48:25.048736 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.048725    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-envoy" pod "18ccc82d-1202-459d-b4f1-fe6fa43d5abf" (UID: "18ccc82d-1202-459d-b4f1-fe6fa43d5abf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.049127 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.049111    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-istio-token" pod "36d86524-a126-407d-85ed-5a62b852046f" (UID: "36d86524-a126-407d-85ed-5a62b852046f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:48:25.049411 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.049392    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-envoy" pod "36d86524-a126-407d-85ed-5a62b852046f" (UID: "36d86524-a126-407d-85ed-5a62b852046f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:48:25.174968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.161467    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.174968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.161504    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5h8d\" (UniqueName: \"kubernetes.io/projected/36d86524-a126-407d-85ed-5a62b852046f-kube-api-access-x5h8d\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.174968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.161520    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.174968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.161535    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36d86524-a126-407d-85ed-5a62b852046f-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.174968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.161548    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.174968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.161562    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.174968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.161575    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrhp5\" (UniqueName: \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-kube-api-access-vrhp5\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.174968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.161588    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36d86524-a126-407d-85ed-5a62b852046f-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.174968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.161601    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18ccc82d-1202-459d-b4f1-fe6fa43d5abf-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:48:25.527971 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.527941    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"]
Oct 30 19:48:25.531961 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.531937    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-nkh6p"]
Oct 30 19:48:25.787166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.775634    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"]
Oct 30 19:48:25.787166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.776719    4420 cpu_manager.go:414] "containerMap: removing container" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy"
Oct 30 19:48:25.787166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.776738    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy"
Oct 30 19:48:25.787166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.776763    4420 cpu_manager.go:414] "containerMap: removing container" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy"
Oct 30 19:48:25.787166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.776773    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy"
Oct 30 19:48:25.787166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.776785    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy"
Oct 30 19:48:25.787166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.776793    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy"
Oct 30 19:48:25.789281 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.789252    4420 cpu_manager.go:414] "containerMap: removing container" podUID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" containerName="istio-proxy"
Oct 30 19:48:25.789281 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.789278    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" containerName="istio-proxy"
Oct 30 19:48:25.789281 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.789292    4420 cpu_manager.go:414] "containerMap: removing container" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerName="istio-proxy"
Oct 30 19:48:25.789522 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.789297    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerName="istio-proxy"
Oct 30 19:48:25.789522 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.789321    4420 cpu_manager.go:414] "containerMap: removing container" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy"
Oct 30 19:48:25.789522 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.789328    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy"
Oct 30 19:48:25.789522 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.789504    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy"
Oct 30 19:48:25.789522 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.789516    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" containerName="istio-proxy"
Oct 30 19:48:25.789522 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.789522    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" containerName="istio-proxy"
Oct 30 19:48:25.789709 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.789531    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5aab9b7-bc7c-44f4-8ec9-b335764484bc" containerName="istio-proxy"
Oct 30 19:48:25.789709 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.789537    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" containerName="istio-proxy"
Oct 30 19:48:25.794300 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.794280    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:25.825149 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.825124    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"]
Oct 30 19:48:25.850165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.850130    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-dd9b8d79-serving-cert\""
Oct 30 19:48:25.857437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.857412    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"
Oct 30 19:48:25.857647 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.857629    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf" event={"ID":"36d86524-a126-407d-85ed-5a62b852046f","Type":"ContainerDied","Data":"e91d3336a42831c73100ad55251fe2470b539b49b8dec42b62a71b6e684180fb"}
Oct 30 19:48:25.857735 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.857720    4420 scope.go:122] "RemoveContainer" containerID="5abebb31940b25d226a8714f7afbc47095350eae061d6d4afa086452cef7ce4f"
Oct 30 19:48:25.887438 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.887405    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5" event={"ID":"570a77ab-2b41-4cb5-8b6d-5231cf2f7c53","Type":"ContainerDied","Data":"cc350211db3598792b983511e9ce027ecdf2d546d49bcde1bbf65b78cde0409c"}
Oct 30 19:48:25.887585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.887531    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"
Oct 30 19:48:25.891188 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.891146    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp" event={"ID":"18ccc82d-1202-459d-b4f1-fe6fa43d5abf","Type":"ContainerDied","Data":"53b879be4247f4442fae453f04900961ca333cd72e9a0dd96e2f521c29b6088f"}
Oct 30 19:48:25.891326 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.891280    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"
Oct 30 19:48:25.893826 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.893689    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b427057-1f04-4da3-a6d7-2d072caf1e12-openshift-service-ca\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:25.894074 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.894050    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxr9d\" (UniqueName: \"kubernetes.io/projected/3b427057-1f04-4da3-a6d7-2d072caf1e12-kube-api-access-sxr9d\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:25.894359 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.894339    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-metrics-certs\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:25.902797 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.902773    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-maas-api-tls\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:25.909116 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.905610    4420 scope.go:122] "RemoveContainer" containerID="030425a44b958edbe52de0ed98d45e513a783ce7e76ae2aaf8b91fdf22889a0e"
Oct 30 19:48:25.926225 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:25.926201    4420 scope.go:122] "RemoveContainer" containerID="7f4b07275c95a464ab0c2a193497a0a3a912fe46c2a106491e03dbf8f79c5326"
Oct 30 19:48:26.022606 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.006989    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-maas-api-tls\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:26.022606 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.007055    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b427057-1f04-4da3-a6d7-2d072caf1e12-openshift-service-ca\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:26.022606 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.007146    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxr9d\" (UniqueName: \"kubernetes.io/projected/3b427057-1f04-4da3-a6d7-2d072caf1e12-kube-api-access-sxr9d\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:26.022606 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.007253    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-metrics-certs\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:26.022606 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:48:26.009432    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-serving-cert: secret "maas-api-e2e-mt-dd9b8d79-serving-cert" not found
Oct 30 19:48:26.022606 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:48:26.009505    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-maas-api-tls podName:3b427057-1f04-4da3-a6d7-2d072caf1e12 nodeName:}" failed. No retries permitted until 2026-10-30 19:48:26.50948561 +0000 UTC m=+1953.720702747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-maas-api-tls") pod "maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9" (UID: "3b427057-1f04-4da3-a6d7-2d072caf1e12") : secret "maas-api-e2e-mt-dd9b8d79-serving-cert" not found
Oct 30 19:48:26.094894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.061222    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"]
Oct 30 19:48:26.094894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.083389    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b427057-1f04-4da3-a6d7-2d072caf1e12-openshift-service-ca\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:26.116888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.116851    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ls2wf"]
Oct 30 19:48:26.135187 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.135155    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-metrics-certs\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:26.165835 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.162267    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxr9d\" (UniqueName: \"kubernetes.io/projected/3b427057-1f04-4da3-a6d7-2d072caf1e12-kube-api-access-sxr9d\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:26.175900 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.175869    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"]
Oct 30 19:48:26.197840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.187693    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-pnks5"]
Oct 30 19:48:26.358897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.357958    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"]
Oct 30 19:48:26.358897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.358015    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a3f9b7-openshift-default-7d6d8b8fb5-ppjmp"]
Oct 30 19:48:26.527931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.527140    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-maas-api-tls\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:26.566478 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.566444    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-maas-api-tls\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:26.709956 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.709925    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:26.938291 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.938259    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ccc82d-1202-459d-b4f1-fe6fa43d5abf" path="/var/lib/kubelet/pods/18ccc82d-1202-459d-b4f1-fe6fa43d5abf/volumes"
Oct 30 19:48:26.939274 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.939251    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d86524-a126-407d-85ed-5a62b852046f" path="/var/lib/kubelet/pods/36d86524-a126-407d-85ed-5a62b852046f/volumes"
Oct 30 19:48:26.939936 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.939918    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570a77ab-2b41-4cb5-8b6d-5231cf2f7c53" path="/var/lib/kubelet/pods/570a77ab-2b41-4cb5-8b6d-5231cf2f7c53/volumes"
Oct 30 19:48:26.940796 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:26.940779    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ffc1f1-605c-4e82-ad64-cc1aee402b79" path="/var/lib/kubelet/pods/a1ffc1f1-605c-4e82-ad64-cc1aee402b79/volumes"
Oct 30 19:48:27.306426 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:27.306169    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"]
Oct 30 19:48:27.988191 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:27.988151    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9" event={"ID":"3b427057-1f04-4da3-a6d7-2d072caf1e12","Type":"ContainerStarted","Data":"e234c0674882c7e8421f03fd3a22ef4001a88b3ebb6659b47f13d0c25c4b90c5"}
Oct 30 19:48:29.068842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:29.064964    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9" event={"ID":"3b427057-1f04-4da3-a6d7-2d072caf1e12","Type":"ContainerStarted","Data":"457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c"}
Oct 30 19:48:29.068842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:29.066097    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:29.184836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:29.184605    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9" podStartSLOduration=3.19624648 podStartE2EDuration="4.184585537s" totalImagesPullingTime="988.339057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:29.182238715 +0000 UTC m=+1956.393455861" watchObservedRunningTime="2026-10-30 19:48:29.184585537 +0000 UTC m=+1956.395802682"
Oct 30 19:48:29.215842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:29.210284    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 30 19:48:29.462554 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:29.461978    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:29.658956 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:29.658913    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded"
Oct 30 19:48:29.667083 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:29.667049    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:48:35.197537 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:48:35.197506    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:48:36.011864 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.011833    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"]
Oct 30 19:48:36.012625 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.012608    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d86524-a126-407d-85ed-5a62b852046f" containerName="istio-proxy"
Oct 30 19:48:36.016173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.016157    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.068598 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.068578    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-6ac304-serving-cert\""
Oct 30 19:48:36.116669 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.116637    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:48:36.124721 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.124697    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"]
Oct 30 19:48:36.133607 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.133584    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f8018e-cf10-47a5-bf36-5e1333e199e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.133719 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.133689    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.133801 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.133782    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.134039 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.134015    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nnf6\" (UniqueName: \"kubernetes.io/projected/d1f8018e-cf10-47a5-bf36-5e1333e199e0-kube-api-access-4nnf6\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.154892 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.154861    4420 generic.go:356] "Generic (PLEG): container finished" podID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerID="099516948db982bb35c82430137ae1c3071fd897659898a3cf5bfdc48c1e3568" exitCode=0
Oct 30 19:48:36.155296 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.155074    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" event={"ID":"7829d2b9-e06d-41c6-98a7-01c17d9b29c8","Type":"ContainerDied","Data":"099516948db982bb35c82430137ae1c3071fd897659898a3cf5bfdc48c1e3568"}
Oct 30 19:48:36.155861 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.155790    4420 scope.go:122] "RemoveContainer" containerID="099516948db982bb35c82430137ae1c3071fd897659898a3cf5bfdc48c1e3568"
Oct 30 19:48:36.158758 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.158722    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:48:36.158898 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.158834    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:48:36.158898 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.158890    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:48:36.235073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.235049    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f8018e-cf10-47a5-bf36-5e1333e199e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.235450 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.235096    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.235450 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.235214    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.235450 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:48:36.235311    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-serving-cert: secret "maas-api-e2e-shared-b-w0-6ac304-serving-cert" not found
Oct 30 19:48:36.235450 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:48:36.235369    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-maas-api-tls podName:d1f8018e-cf10-47a5-bf36-5e1333e199e0 nodeName:}" failed. No retries permitted until 2026-10-30 19:48:36.735349305 +0000 UTC m=+1963.946566455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-maas-api-tls") pod "maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr" (UID: "d1f8018e-cf10-47a5-bf36-5e1333e199e0") : secret "maas-api-e2e-shared-b-w0-6ac304-serving-cert" not found
Oct 30 19:48:36.235690 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.235492    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nnf6\" (UniqueName: \"kubernetes.io/projected/d1f8018e-cf10-47a5-bf36-5e1333e199e0-kube-api-access-4nnf6\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.235879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.235861    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f8018e-cf10-47a5-bf36-5e1333e199e0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.238072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.238052    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.333418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.333332    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nnf6\" (UniqueName: \"kubernetes.io/projected/d1f8018e-cf10-47a5-bf36-5e1333e199e0-kube-api-access-4nnf6\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.550901 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.550357    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:48:36.751385 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.751347    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.754105 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.754080    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:36.930640 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:36.930612    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:37.188659 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:37.188623    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" event={"ID":"7829d2b9-e06d-41c6-98a7-01c17d9b29c8","Type":"ContainerStarted","Data":"32c36ecacbd0f05b2bd5c16c1e995f508c9213c5b850efb525bae65ba45790fe"}
Oct 30 19:48:37.342656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:37.342579    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"]
Oct 30 19:48:37.347600 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:48:37.347565    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f8018e_cf10_47a5_bf36_5e1333e199e0.slice/crio-945e998ffe6e828aabe22dfe5834a898bc72fd5aaef352b687533f24a3c3f000 WatchSource:0}: Error finding container 945e998ffe6e828aabe22dfe5834a898bc72fd5aaef352b687533f24a3c3f000: Status 404 returned error can't find the container with id 945e998ffe6e828aabe22dfe5834a898bc72fd5aaef352b687533f24a3c3f000
Oct 30 19:48:37.894611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:37.894577    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:48:37.895745 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:37.895719    4420 patch_prober.go:29] interesting pod/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:48:37.895900 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:37.895768    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:48:38.201069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:38.200968    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr" event={"ID":"d1f8018e-cf10-47a5-bf36-5e1333e199e0","Type":"ContainerStarted","Data":"945e998ffe6e828aabe22dfe5834a898bc72fd5aaef352b687533f24a3c3f000"}
Oct 30 19:48:38.894872 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:38.894832    4420 patch_prober.go:29] interesting pod/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:48:38.895267 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:38.894907    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:48:39.210547 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:39.210457    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr" event={"ID":"d1f8018e-cf10-47a5-bf36-5e1333e199e0","Type":"ContainerStarted","Data":"4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253"}
Oct 30 19:48:39.210710 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:39.210629    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:48:39.328173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:39.328117    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr" podStartSLOduration=3.286544717 podStartE2EDuration="4.328102962s" totalImagesPullingTime="1.041558245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:39.327420456 +0000 UTC m=+1966.538637600" watchObservedRunningTime="2026-10-30 19:48:39.328102962 +0000 UTC m=+1966.539320139"
Oct 30 19:48:39.894774 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:39.894740    4420 patch_prober.go:29] interesting pod/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:48:39.894978 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:39.894810    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:48:40.894794 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:40.894747    4420 patch_prober.go:29] interesting pod/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:48:40.895051 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:40.894847    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:48:41.895165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:41.895137    4420 patch_prober.go:29] interesting pod/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 30 19:48:41.895656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:41.895186    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 30 19:48:42.899810 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:42.899783    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:48:42.900759 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:42.900715    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:48:42.901135 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:42.901119    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:48:45.222803 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:48:45.222777    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 19:49:05.197724 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:49:05.197687    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:49:09.936610 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:09.936575    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"]
Oct 30 19:49:09.937017 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:09.936786    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy" containerID="cri-o://4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767" gracePeriod=30
Oct 30 19:49:15.230426 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.230401    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:49:15.306969 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.306935    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-credential-socket\") pod \"d0379827-bfec-4f2c-86b0-fb38be8d6281\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") "
Oct 30 19:49:15.307130 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.306987    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-certs\") pod \"d0379827-bfec-4f2c-86b0-fb38be8d6281\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") "
Oct 30 19:49:15.307130 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307025    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-envoy\") pod \"d0379827-bfec-4f2c-86b0-fb38be8d6281\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") "
Oct 30 19:49:15.307130 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307124    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-token\" (UniqueName: \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-token\") pod \"d0379827-bfec-4f2c-86b0-fb38be8d6281\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") "
Oct 30 19:49:15.307250 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307171    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0379827-bfec-4f2c-86b0-fb38be8d6281-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0379827-bfec-4f2c-86b0-fb38be8d6281-istiod-ca-cert\") pod \"d0379827-bfec-4f2c-86b0-fb38be8d6281\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") "
Oct 30 19:49:15.307250 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307172    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-credential-socket" pod "d0379827-bfec-4f2c-86b0-fb38be8d6281" (UID: "d0379827-bfec-4f2c-86b0-fb38be8d6281"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.307250 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307229    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-podinfo\") pod \"d0379827-bfec-4f2c-86b0-fb38be8d6281\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") "
Oct 30 19:49:15.307381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307261    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-data\") pod \"d0379827-bfec-4f2c-86b0-fb38be8d6281\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") "
Oct 30 19:49:15.307381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307309    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-certs" pod "d0379827-bfec-4f2c-86b0-fb38be8d6281" (UID: "d0379827-bfec-4f2c-86b0-fb38be8d6281"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.307489 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307387    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-kube-api-access-26p7r\" (UniqueName: \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-kube-api-access-26p7r\") pod \"d0379827-bfec-4f2c-86b0-fb38be8d6281\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") "
Oct 30 19:49:15.307572 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307543    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-data" pod "d0379827-bfec-4f2c-86b0-fb38be8d6281" (UID: "d0379827-bfec-4f2c-86b0-fb38be8d6281"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.307572 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307556    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-socket\") pod \"d0379827-bfec-4f2c-86b0-fb38be8d6281\" (UID: \"d0379827-bfec-4f2c-86b0-fb38be8d6281\") "
Oct 30 19:49:15.307684 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307620    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0379827-bfec-4f2c-86b0-fb38be8d6281-istiod-ca-cert" pod "d0379827-bfec-4f2c-86b0-fb38be8d6281" (UID: "d0379827-bfec-4f2c-86b0-fb38be8d6281"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:15.307891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.307859    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-socket" pod "d0379827-bfec-4f2c-86b0-fb38be8d6281" (UID: "d0379827-bfec-4f2c-86b0-fb38be8d6281"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.308508 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.308483    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.308612 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.308513    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0379827-bfec-4f2c-86b0-fb38be8d6281-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.308612 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.308529    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.308612 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.308541    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.308612 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.308555    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.309545 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.309515    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-envoy" pod "d0379827-bfec-4f2c-86b0-fb38be8d6281" (UID: "d0379827-bfec-4f2c-86b0-fb38be8d6281"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:15.309545 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.309521    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-token" pod "d0379827-bfec-4f2c-86b0-fb38be8d6281" (UID: "d0379827-bfec-4f2c-86b0-fb38be8d6281"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.309687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.309586    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-podinfo" pod "d0379827-bfec-4f2c-86b0-fb38be8d6281" (UID: "d0379827-bfec-4f2c-86b0-fb38be8d6281"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:15.309726 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.309681    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-kube-api-access-26p7r" pod "d0379827-bfec-4f2c-86b0-fb38be8d6281" (UID: "d0379827-bfec-4f2c-86b0-fb38be8d6281"). InnerVolumeSpecName "kube-api-access-26p7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:15.410769 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.410707    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.410769 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.410744    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.410769 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.410758    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0379827-bfec-4f2c-86b0-fb38be8d6281-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.413339 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.410774    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26p7r\" (UniqueName: \"kubernetes.io/projected/d0379827-bfec-4f2c-86b0-fb38be8d6281-kube-api-access-26p7r\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:15.479436 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.479353    4420 generic.go:356] "Generic (PLEG): container finished" podID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerID="4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767" exitCode=0
Oct 30 19:49:15.479436 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.479401    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"
Oct 30 19:49:15.479629 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.479438    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" event={"ID":"d0379827-bfec-4f2c-86b0-fb38be8d6281","Type":"ContainerDied","Data":"4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767"}
Oct 30 19:49:15.479629 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.479494    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz" event={"ID":"d0379827-bfec-4f2c-86b0-fb38be8d6281","Type":"ContainerDied","Data":"c24c80016f336fa08a8b47e45ef88310dd0164a6c2a3be23dc48f000fa23d09b"}
Oct 30 19:49:15.479629 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.479515    4420 scope.go:122] "RemoveContainer" containerID="4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767"
Oct 30 19:49:15.492049 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.492029    4420 scope.go:122] "RemoveContainer" containerID="4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767"
Oct 30 19:49:15.492386 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:49:15.492349    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767\": container with ID starting with 4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767 not found: ID does not exist" containerID="4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767"
Oct 30 19:49:15.492461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.492400    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767"} err="failed to get container status \"4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767\": rpc error: code = NotFound desc = could not find container \"4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767\": container with ID starting with 4587d2d2b727440bbe4abd0df1bd3a97603614b8a77bdac21664bf6254091767 not found: ID does not exist"
Oct 30 19:49:15.651493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.651464    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"]
Oct 30 19:49:15.654975 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:15.654949    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-dcgpz"]
Oct 30 19:49:16.897688 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:16.897656    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" path="/var/lib/kubelet/pods/d0379827-bfec-4f2c-86b0-fb38be8d6281/volumes"
Oct 30 19:49:35.201521 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:49:35.201440    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:49:40.737326 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:40.737296    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"]
Oct 30 19:49:40.737797 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:40.737504    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy" containerID="cri-o://989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1" gracePeriod=30
Oct 30 19:49:40.788073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:40.788045    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"]
Oct 30 19:49:40.788258 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:40.788239    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy" containerID="cri-o://6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb" gracePeriod=30
Oct 30 19:49:40.791939 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:40.791915    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"]
Oct 30 19:49:40.792121 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:40.792087    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy" containerID="cri-o://a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd" gracePeriod=30
Oct 30 19:49:40.836405 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:40.836375    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"]
Oct 30 19:49:40.836575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:40.836554    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy" containerID="cri-o://211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7" gracePeriod=30
Oct 30 19:49:42.512433 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:42.512398    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"]
Oct 30 19:49:42.512891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:42.512664    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" podUID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" containerName="istio-proxy" containerID="cri-o://613c61d3c390c4458a616688595ae8843331d58b2ad024f12f0fae0caae05605" gracePeriod=30
Oct 30 19:49:42.560478 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:42.560449    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"]
Oct 30 19:49:42.560677 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:42.560657    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy" containerID="cri-o://761902ef60a86f2d4d82b8050ca9bfd3e6563820396f99491ceaa313bc728c9d" gracePeriod=30
Oct 30 19:49:42.564340 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:42.564317    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"]
Oct 30 19:49:42.609113 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:42.609085    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"]
Oct 30 19:49:42.609526 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:42.609491    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" podUID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" containerName="istio-proxy" containerID="cri-o://f74af44b3e78ee7bab8d0e0be426f0bf4e76df8f6c6be2e238cff2fc86dfb9bc" gracePeriod=30
Oct 30 19:49:42.671620 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:42.671584    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" podUID="7c160df8-7210-45d5-b6bb-d42ae62c5310" containerName="istio-proxy" containerID="cri-o://7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4" gracePeriod=30
Oct 30 19:49:43.246455 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:43.246417    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:43.246623 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:43.246487    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:44.979722 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:44.979692    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:44.980192 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:44.979746    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:45.080387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:45.080355    4420 patch_prober.go:29] interesting pod/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:45.080544 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:45.080428    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:45.104986 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:45.104960    4420 patch_prober.go:29] interesting pod/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:49:45.105107 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:45.105005    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:49:46.120329 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.120301    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:49:46.155123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.153956    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-certs\") pod \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") "
Oct 30 19:49:46.155123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.154048    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-token\" (UniqueName: \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-token\") pod \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") "
Oct 30 19:49:46.155123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.154098    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-data\") pod \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") "
Oct 30 19:49:46.155123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.154249    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-socket\") pod \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") "
Oct 30 19:49:46.155123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.154312    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-envoy\") pod \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") "
Oct 30 19:49:46.155123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.154371    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-podinfo\") pod \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") "
Oct 30 19:49:46.155123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.154407    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istiod-ca-cert\") pod \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") "
Oct 30 19:49:46.155123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.154463    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-credential-socket\") pod \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") "
Oct 30 19:49:46.155123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.154491    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-kube-api-access-266vr\" (UniqueName: \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-kube-api-access-266vr\") pod \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\" (UID: \"bfde16f9-2bea-4cf0-8c3f-7e689934de26\") "
Oct 30 19:49:46.155123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.154940    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-socket" pod "bfde16f9-2bea-4cf0-8c3f-7e689934de26" (UID: "bfde16f9-2bea-4cf0-8c3f-7e689934de26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.155123 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.154888    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-data" pod "bfde16f9-2bea-4cf0-8c3f-7e689934de26" (UID: "bfde16f9-2bea-4cf0-8c3f-7e689934de26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.155877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.155298    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istiod-ca-cert" pod "bfde16f9-2bea-4cf0-8c3f-7e689934de26" (UID: "bfde16f9-2bea-4cf0-8c3f-7e689934de26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:46.155877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.155690    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.155877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.155712    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.155877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.155728    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.156142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.155994    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-certs" pod "bfde16f9-2bea-4cf0-8c3f-7e689934de26" (UID: "bfde16f9-2bea-4cf0-8c3f-7e689934de26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.156320 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.156288    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-credential-socket" pod "bfde16f9-2bea-4cf0-8c3f-7e689934de26" (UID: "bfde16f9-2bea-4cf0-8c3f-7e689934de26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.158733 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.158712    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-token" pod "bfde16f9-2bea-4cf0-8c3f-7e689934de26" (UID: "bfde16f9-2bea-4cf0-8c3f-7e689934de26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.159633 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.159597    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-envoy" pod "bfde16f9-2bea-4cf0-8c3f-7e689934de26" (UID: "bfde16f9-2bea-4cf0-8c3f-7e689934de26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.165560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.165534    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-kube-api-access-266vr" pod "bfde16f9-2bea-4cf0-8c3f-7e689934de26" (UID: "bfde16f9-2bea-4cf0-8c3f-7e689934de26"). InnerVolumeSpecName "kube-api-access-266vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.166481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.166405    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:49:46.166582 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.166473    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-podinfo" pod "bfde16f9-2bea-4cf0-8c3f-7e689934de26" (UID: "bfde16f9-2bea-4cf0-8c3f-7e689934de26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:46.169722 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.169703    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:49:46.203409 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.203386    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:49:46.266487 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.259362    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.266487 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.259399    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.266487 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.259415    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfde16f9-2bea-4cf0-8c3f-7e689934de26-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.266487 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.259429    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.266487 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.259444    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-266vr\" (UniqueName: \"kubernetes.io/projected/bfde16f9-2bea-4cf0-8c3f-7e689934de26-kube-api-access-266vr\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.266487 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.259461    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfde16f9-2bea-4cf0-8c3f-7e689934de26-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.365659 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.365522    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-data\") pod \"9813d487-1f24-47f9-9cea-47e54083d91e\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") "
Oct 30 19:49:46.366002 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.365967    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b0996b-c55b-4e37-8220-1302701f6260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b0996b-c55b-4e37-8220-1302701f6260-istiod-ca-cert\") pod \"29b0996b-c55b-4e37-8220-1302701f6260\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") "
Oct 30 19:49:46.366264 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.366228    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-credential-socket\") pod \"9813d487-1f24-47f9-9cea-47e54083d91e\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") "
Oct 30 19:49:46.366490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.366459    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-certs\") pod \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") "
Oct 30 19:49:46.366706 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.366663    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-credential-socket" pod "9813d487-1f24-47f9-9cea-47e54083d91e" (UID: "9813d487-1f24-47f9-9cea-47e54083d91e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.366879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.366860    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-socket\") pod \"29b0996b-c55b-4e37-8220-1302701f6260\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") "
Oct 30 19:49:46.367029 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.367013    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-kube-api-access-mnl79\" (UniqueName: \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-kube-api-access-mnl79\") pod \"9813d487-1f24-47f9-9cea-47e54083d91e\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") "
Oct 30 19:49:46.367152 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.367136    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-certs\") pod \"9813d487-1f24-47f9-9cea-47e54083d91e\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") "
Oct 30 19:49:46.367309 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.367295    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-socket\") pod \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") "
Oct 30 19:49:46.367406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.367393    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9813d487-1f24-47f9-9cea-47e54083d91e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9813d487-1f24-47f9-9cea-47e54083d91e-istiod-ca-cert\") pod \"9813d487-1f24-47f9-9cea-47e54083d91e\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") "
Oct 30 19:49:46.367509 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.367496    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-data\") pod \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") "
Oct 30 19:49:46.367599 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.367587    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istiod-ca-cert\") pod \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") "
Oct 30 19:49:46.367873 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.367798    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-data\") pod \"29b0996b-c55b-4e37-8220-1302701f6260\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") "
Oct 30 19:49:46.368160 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.368142    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-envoy\") pod \"9813d487-1f24-47f9-9cea-47e54083d91e\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") "
Oct 30 19:49:46.368286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.368272    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9813d487-1f24-47f9-9cea-47e54083d91e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9813d487-1f24-47f9-9cea-47e54083d91e-istio-podinfo\") pod \"9813d487-1f24-47f9-9cea-47e54083d91e\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") "
Oct 30 19:49:46.368402 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.368389    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-istio-token\" (UniqueName: \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-istio-token\") pod \"9813d487-1f24-47f9-9cea-47e54083d91e\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") "
Oct 30 19:49:46.368516 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.368503    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-envoy\") pod \"29b0996b-c55b-4e37-8220-1302701f6260\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") "
Oct 30 19:49:46.368653 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.368637    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-podinfo\") pod \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") "
Oct 30 19:49:46.368799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.368784    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-socket\") pod \"9813d487-1f24-47f9-9cea-47e54083d91e\" (UID: \"9813d487-1f24-47f9-9cea-47e54083d91e\") "
Oct 30 19:49:46.368962 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.368944    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-credential-socket\") pod \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") "
Oct 30 19:49:46.369078 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.369063    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-kube-api-access-mgb6j\" (UniqueName: \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-kube-api-access-mgb6j\") pod \"29b0996b-c55b-4e37-8220-1302701f6260\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") "
Oct 30 19:49:46.369227 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.369211    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-credential-socket\") pod \"29b0996b-c55b-4e37-8220-1302701f6260\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") "
Oct 30 19:49:46.369368 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.369353    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-certs\") pod \"29b0996b-c55b-4e37-8220-1302701f6260\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") "
Oct 30 19:49:46.369486 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.369473    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-envoy\") pod \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") "
Oct 30 19:49:46.369590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.369577    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-istio-token\" (UniqueName: \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-istio-token\") pod \"29b0996b-c55b-4e37-8220-1302701f6260\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") "
Oct 30 19:49:46.369704 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.369690    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29b0996b-c55b-4e37-8220-1302701f6260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b0996b-c55b-4e37-8220-1302701f6260-istio-podinfo\") pod \"29b0996b-c55b-4e37-8220-1302701f6260\" (UID: \"29b0996b-c55b-4e37-8220-1302701f6260\") "
Oct 30 19:49:46.369860 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.369845    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-token\" (UniqueName: \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-token\") pod \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") "
Oct 30 19:49:46.370173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.366928    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-certs" pod "8e0b55de-ce41-492a-846d-bd4533cfa6cc" (UID: "8e0b55de-ce41-492a-846d-bd4533cfa6cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.370173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.367122    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-socket" pod "29b0996b-c55b-4e37-8220-1302701f6260" (UID: "29b0996b-c55b-4e37-8220-1302701f6260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.370173 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.368204    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9813d487-1f24-47f9-9cea-47e54083d91e-istiod-ca-cert" pod "9813d487-1f24-47f9-9cea-47e54083d91e" (UID: "9813d487-1f24-47f9-9cea-47e54083d91e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:46.370357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.368432    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-socket" pod "8e0b55de-ce41-492a-846d-bd4533cfa6cc" (UID: "8e0b55de-ce41-492a-846d-bd4533cfa6cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.370357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.368716    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-data" pod "8e0b55de-ce41-492a-846d-bd4533cfa6cc" (UID: "8e0b55de-ce41-492a-846d-bd4533cfa6cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.370357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.368965    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-data" pod "29b0996b-c55b-4e37-8220-1302701f6260" (UID: "29b0996b-c55b-4e37-8220-1302701f6260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.370357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.369298    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istiod-ca-cert" pod "8e0b55de-ce41-492a-846d-bd4533cfa6cc" (UID: "8e0b55de-ce41-492a-846d-bd4533cfa6cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:46.370357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.369541    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-certs" pod "9813d487-1f24-47f9-9cea-47e54083d91e" (UID: "9813d487-1f24-47f9-9cea-47e54083d91e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.370357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.369796    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-credential-socket" pod "8e0b55de-ce41-492a-846d-bd4533cfa6cc" (UID: "8e0b55de-ce41-492a-846d-bd4533cfa6cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.370357 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.370107    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b0996b-c55b-4e37-8220-1302701f6260-istiod-ca-cert" pod "29b0996b-c55b-4e37-8220-1302701f6260" (UID: "29b0996b-c55b-4e37-8220-1302701f6260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:46.370881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.370861    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-kube-api-access-zgr2m\" (UniqueName: \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-kube-api-access-zgr2m\") pod \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\" (UID: \"8e0b55de-ce41-492a-846d-bd4533cfa6cc\") "
Oct 30 19:49:46.371423 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.371374    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-data" pod "9813d487-1f24-47f9-9cea-47e54083d91e" (UID: "9813d487-1f24-47f9-9cea-47e54083d91e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.372195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.371796    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-certs" pod "29b0996b-c55b-4e37-8220-1302701f6260" (UID: "29b0996b-c55b-4e37-8220-1302701f6260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.377863 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.377581    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-socket" pod "9813d487-1f24-47f9-9cea-47e54083d91e" (UID: "9813d487-1f24-47f9-9cea-47e54083d91e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.378695 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.378209    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-credential-socket" pod "29b0996b-c55b-4e37-8220-1302701f6260" (UID: "29b0996b-c55b-4e37-8220-1302701f6260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379197    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379224    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379240    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379254    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379267    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379279    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379293    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29b0996b-c55b-4e37-8220-1302701f6260-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379306    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379318    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379331    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379344    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379358    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379370    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9813d487-1f24-47f9-9cea-47e54083d91e-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379383    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.379425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.379396    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.383828 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.383763    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29b0996b-c55b-4e37-8220-1302701f6260-istio-podinfo" pod "29b0996b-c55b-4e37-8220-1302701f6260" (UID: "29b0996b-c55b-4e37-8220-1302701f6260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:46.384603 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.384562    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-envoy" pod "8e0b55de-ce41-492a-846d-bd4533cfa6cc" (UID: "8e0b55de-ce41-492a-846d-bd4533cfa6cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.384930 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.384896    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-kube-api-access-mgb6j" pod "29b0996b-c55b-4e37-8220-1302701f6260" (UID: "29b0996b-c55b-4e37-8220-1302701f6260"). InnerVolumeSpecName "kube-api-access-mgb6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.385192 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.385165    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-istio-token" pod "9813d487-1f24-47f9-9cea-47e54083d91e" (UID: "9813d487-1f24-47f9-9cea-47e54083d91e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.385381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.385365    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-istio-token" pod "29b0996b-c55b-4e37-8220-1302701f6260" (UID: "29b0996b-c55b-4e37-8220-1302701f6260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.385570 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.385524    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-kube-api-access-zgr2m" pod "8e0b55de-ce41-492a-846d-bd4533cfa6cc" (UID: "8e0b55de-ce41-492a-846d-bd4533cfa6cc"). InnerVolumeSpecName "kube-api-access-zgr2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.385784 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.385748    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-podinfo" pod "8e0b55de-ce41-492a-846d-bd4533cfa6cc" (UID: "8e0b55de-ce41-492a-846d-bd4533cfa6cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:46.385784 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.385771    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9813d487-1f24-47f9-9cea-47e54083d91e-istio-podinfo" pod "9813d487-1f24-47f9-9cea-47e54083d91e" (UID: "9813d487-1f24-47f9-9cea-47e54083d91e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:46.386234 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.386011    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-envoy" pod "29b0996b-c55b-4e37-8220-1302701f6260" (UID: "29b0996b-c55b-4e37-8220-1302701f6260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.386234 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.386082    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-envoy" pod "9813d487-1f24-47f9-9cea-47e54083d91e" (UID: "9813d487-1f24-47f9-9cea-47e54083d91e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:46.386234 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.386198    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-kube-api-access-mnl79" pod "9813d487-1f24-47f9-9cea-47e54083d91e" (UID: "9813d487-1f24-47f9-9cea-47e54083d91e"). InnerVolumeSpecName "kube-api-access-mnl79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.388011 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.387977    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-token" pod "8e0b55de-ce41-492a-846d-bd4533cfa6cc" (UID: "8e0b55de-ce41-492a-846d-bd4533cfa6cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:46.480626 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480596    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9813d487-1f24-47f9-9cea-47e54083d91e-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.480626 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480624    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9813d487-1f24-47f9-9cea-47e54083d91e-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.480626 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480635    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.480878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480643    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29b0996b-c55b-4e37-8220-1302701f6260-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.480878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480651    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.480878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480660    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgb6j\" (UniqueName: \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-kube-api-access-mgb6j\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.480878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480669    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.480878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480677    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29b0996b-c55b-4e37-8220-1302701f6260-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.480878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480697    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29b0996b-c55b-4e37-8220-1302701f6260-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.480878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480706    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.480878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480715    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgr2m\" (UniqueName: \"kubernetes.io/projected/8e0b55de-ce41-492a-846d-bd4533cfa6cc-kube-api-access-zgr2m\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.480878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.480723    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnl79\" (UniqueName: \"kubernetes.io/projected/9813d487-1f24-47f9-9cea-47e54083d91e-kube-api-access-mnl79\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:46.702237 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.702198    4420 generic.go:356] "Generic (PLEG): container finished" podID="9813d487-1f24-47f9-9cea-47e54083d91e" containerID="6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb" exitCode=0
Oct 30 19:49:46.702434 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.702260    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"
Oct 30 19:49:46.702434 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.702279    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" event={"ID":"9813d487-1f24-47f9-9cea-47e54083d91e","Type":"ContainerDied","Data":"6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb"}
Oct 30 19:49:46.702434 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.702340    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt" event={"ID":"9813d487-1f24-47f9-9cea-47e54083d91e","Type":"ContainerDied","Data":"6ce740186cfc85ba041fa9bd5f52d3adff33da0a9e6af28d9e00b5a5668b7b0a"}
Oct 30 19:49:46.702434 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.702369    4420 scope.go:122] "RemoveContainer" containerID="6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb"
Oct 30 19:49:46.704243 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.704217    4420 generic.go:356] "Generic (PLEG): container finished" podID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerID="a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd" exitCode=0
Oct 30 19:49:46.704541 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.704523    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" event={"ID":"8e0b55de-ce41-492a-846d-bd4533cfa6cc","Type":"ContainerDied","Data":"a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd"}
Oct 30 19:49:46.704624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.704569    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq" event={"ID":"8e0b55de-ce41-492a-846d-bd4533cfa6cc","Type":"ContainerDied","Data":"580781aeb4627a38e1fe9d3b076a4ebe38199062a9f45b7365aa5d1e23b88f12"}
Oct 30 19:49:46.704692 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.704641    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"
Oct 30 19:49:46.715858 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.715834    4420 generic.go:356] "Generic (PLEG): container finished" podID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerID="989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1" exitCode=0
Oct 30 19:49:46.716001 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.715988    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"
Oct 30 19:49:46.716236 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.716198    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" event={"ID":"bfde16f9-2bea-4cf0-8c3f-7e689934de26","Type":"ContainerDied","Data":"989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1"}
Oct 30 19:49:46.716868 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.716846    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt" event={"ID":"bfde16f9-2bea-4cf0-8c3f-7e689934de26","Type":"ContainerDied","Data":"ede054ce7e5d2c5473f8084f773b409150555483f0d4fc901f123007bd279a13"}
Oct 30 19:49:46.718789 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.718767    4420 scope.go:122] "RemoveContainer" containerID="6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb"
Oct 30 19:49:46.718931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.718887    4420 generic.go:356] "Generic (PLEG): container finished" podID="29b0996b-c55b-4e37-8220-1302701f6260" containerID="211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7" exitCode=0
Oct 30 19:49:46.718931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.718920    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" event={"ID":"29b0996b-c55b-4e37-8220-1302701f6260","Type":"ContainerDied","Data":"211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7"}
Oct 30 19:49:46.719018 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.718954    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"
Oct 30 19:49:46.719116 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:49:46.719095    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb\": container with ID starting with 6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb not found: ID does not exist" containerID="6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb"
Oct 30 19:49:46.719162 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.719127    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb"} err="failed to get container status \"6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb\": rpc error: code = NotFound desc = could not find container \"6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb\": container with ID starting with 6923162a613eb060efe915921bb5e1bb76f435e753833330c0c5a0b6a35bcffb not found: ID does not exist"
Oct 30 19:49:46.719162 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.719146    4420 scope.go:122] "RemoveContainer" containerID="a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd"
Oct 30 19:49:46.719287 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.718955    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5" event={"ID":"29b0996b-c55b-4e37-8220-1302701f6260","Type":"ContainerDied","Data":"e21fefaca12136b95399b84c87fa15f1832b0b5f3181fc43965aab88c6d0e85c"}
Oct 30 19:49:46.731993 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.731805    4420 scope.go:122] "RemoveContainer" containerID="a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd"
Oct 30 19:49:46.732274 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:49:46.732254    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd\": container with ID starting with a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd not found: ID does not exist" containerID="a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd"
Oct 30 19:49:46.732339 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.732284    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd"} err="failed to get container status \"a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd\": rpc error: code = NotFound desc = could not find container \"a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd\": container with ID starting with a3ecbbd7041decc63161c24b8a353a7e58b88a1759324b45a7a8c265eda7f5dd not found: ID does not exist"
Oct 30 19:49:46.732339 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.732307    4420 scope.go:122] "RemoveContainer" containerID="989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1"
Oct 30 19:49:46.741525 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.741498    4420 scope.go:122] "RemoveContainer" containerID="989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1"
Oct 30 19:49:46.741778 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:49:46.741758    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1\": container with ID starting with 989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1 not found: ID does not exist" containerID="989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1"
Oct 30 19:49:46.741899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.741782    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1"} err="failed to get container status \"989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1\": rpc error: code = NotFound desc = could not find container \"989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1\": container with ID starting with 989e8c8a6dafdb18b08f79c806af4e05313e8a709f99a2283b4cbc78324aa5a1 not found: ID does not exist"
Oct 30 19:49:46.741899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.741798    4420 scope.go:122] "RemoveContainer" containerID="211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7"
Oct 30 19:49:46.750520 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.750491    4420 scope.go:122] "RemoveContainer" containerID="69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945"
Oct 30 19:49:46.759331 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.759305    4420 scope.go:122] "RemoveContainer" containerID="211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7"
Oct 30 19:49:46.759605 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:49:46.759587    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7\": container with ID starting with 211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7 not found: ID does not exist" containerID="211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7"
Oct 30 19:49:46.759656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.759612    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7"} err="failed to get container status \"211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7\": rpc error: code = NotFound desc = could not find container \"211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7\": container with ID starting with 211e308eef79499735849c32aca417de20db6f3d56fa9fb36e7f28529c13ffc7 not found: ID does not exist"
Oct 30 19:49:46.759656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.759627    4420 scope.go:122] "RemoveContainer" containerID="69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945"
Oct 30 19:49:46.759877 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:49:46.759860    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945\": container with ID starting with 69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945 not found: ID does not exist" containerID="69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945"
Oct 30 19:49:46.759930 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.759885    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945"} err="failed to get container status \"69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945\": rpc error: code = NotFound desc = could not find container \"69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945\": container with ID starting with 69565541b8a82ce859ea61cff45306e120f60602f179c68571527d5f0db5a945 not found: ID does not exist"
Oct 30 19:49:46.867473 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.867441    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"]
Oct 30 19:49:46.871586 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.871560    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pp4v5"]
Oct 30 19:49:46.897669 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.897641    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b0996b-c55b-4e37-8220-1302701f6260" path="/var/lib/kubelet/pods/29b0996b-c55b-4e37-8220-1302701f6260/volumes"
Oct 30 19:49:46.988347 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.988267    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"]
Oct 30 19:49:46.992716 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:46.992690    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-25hkt"]
Oct 30 19:49:47.091338 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.091308    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"]
Oct 30 19:49:47.094634 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.094609    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-pvqqt"]
Oct 30 19:49:47.202433 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.202404    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"]
Oct 30 19:49:47.211241 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.211214    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d52df0-openshift-default-6d5f887b45-t5cqq"]
Oct 30 19:49:47.727617 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.727591    4420 generic.go:356] "Generic (PLEG): container finished" podID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" containerID="613c61d3c390c4458a616688595ae8843331d58b2ad024f12f0fae0caae05605" exitCode=0
Oct 30 19:49:47.727768 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.727668    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" event={"ID":"7b4d6af5-bc40-4313-b009-2c1d61ee9eab","Type":"ContainerDied","Data":"613c61d3c390c4458a616688595ae8843331d58b2ad024f12f0fae0caae05605"}
Oct 30 19:49:47.742135 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.741898    4420 generic.go:356] "Generic (PLEG): container finished" podID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerID="761902ef60a86f2d4d82b8050ca9bfd3e6563820396f99491ceaa313bc728c9d" exitCode=0
Oct 30 19:49:47.742135 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.741987    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" event={"ID":"90a417e0-9ddc-4187-aaf8-632e3e20401f","Type":"ContainerDied","Data":"761902ef60a86f2d4d82b8050ca9bfd3e6563820396f99491ceaa313bc728c9d"}
Oct 30 19:49:47.742135 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.742041    4420 scope.go:122] "RemoveContainer" containerID="f5ea3b5ae48e1b6d63ab68f2a812ff11c23b88d287e4ee723bc0be585595e260"
Oct 30 19:49:47.753656 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.753606    4420 generic.go:356] "Generic (PLEG): container finished" podID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" containerID="f74af44b3e78ee7bab8d0e0be426f0bf4e76df8f6c6be2e238cff2fc86dfb9bc" exitCode=0
Oct 30 19:49:47.753781 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.753758    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" event={"ID":"3751f391-d63e-489b-9ae9-8ad55dd4f1b2","Type":"ContainerDied","Data":"f74af44b3e78ee7bab8d0e0be426f0bf4e76df8f6c6be2e238cff2fc86dfb9bc"}
Oct 30 19:49:47.881313 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.880984    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:49:47.913703 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.913586    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-token\" (UniqueName: \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-token\") pod \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") "
Oct 30 19:49:47.913912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.913714    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-certs\") pod \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") "
Oct 30 19:49:47.913912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.913779    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istiod-ca-cert\") pod \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") "
Oct 30 19:49:47.914038 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.914011    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-socket\") pod \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") "
Oct 30 19:49:47.914100 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.914063    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-data\") pod \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") "
Oct 30 19:49:47.914161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.914097    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-envoy\") pod \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") "
Oct 30 19:49:47.914161 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.914134    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-kube-api-access-wf8tl\" (UniqueName: \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-kube-api-access-wf8tl\") pod \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") "
Oct 30 19:49:47.914259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.914180    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-podinfo\") pod \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") "
Oct 30 19:49:47.914305 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.914287    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-credential-socket\") pod \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\" (UID: \"7b4d6af5-bc40-4313-b009-2c1d61ee9eab\") "
Oct 30 19:49:47.919214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.916079    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-certs" pod "7b4d6af5-bc40-4313-b009-2c1d61ee9eab" (UID: "7b4d6af5-bc40-4313-b009-2c1d61ee9eab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:47.919214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.916362    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istiod-ca-cert" pod "7b4d6af5-bc40-4313-b009-2c1d61ee9eab" (UID: "7b4d6af5-bc40-4313-b009-2c1d61ee9eab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:47.919214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.916586    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-socket" pod "7b4d6af5-bc40-4313-b009-2c1d61ee9eab" (UID: "7b4d6af5-bc40-4313-b009-2c1d61ee9eab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:47.919214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.916754    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-data" pod "7b4d6af5-bc40-4313-b009-2c1d61ee9eab" (UID: "7b4d6af5-bc40-4313-b009-2c1d61ee9eab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:47.920398 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.920190    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-envoy" pod "7b4d6af5-bc40-4313-b009-2c1d61ee9eab" (UID: "7b4d6af5-bc40-4313-b009-2c1d61ee9eab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:47.920613 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.920577    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-credential-socket" pod "7b4d6af5-bc40-4313-b009-2c1d61ee9eab" (UID: "7b4d6af5-bc40-4313-b009-2c1d61ee9eab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:47.922187 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.922136    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-token" pod "7b4d6af5-bc40-4313-b009-2c1d61ee9eab" (UID: "7b4d6af5-bc40-4313-b009-2c1d61ee9eab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:47.922887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.922846    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-kube-api-access-wf8tl" pod "7b4d6af5-bc40-4313-b009-2c1d61ee9eab" (UID: "7b4d6af5-bc40-4313-b009-2c1d61ee9eab"). InnerVolumeSpecName "kube-api-access-wf8tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:47.934506 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:47.934413    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-podinfo" pod "7b4d6af5-bc40-4313-b009-2c1d61ee9eab" (UID: "7b4d6af5-bc40-4313-b009-2c1d61ee9eab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:48.017475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.017322    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.017475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.017354    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.017475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.017369    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.017475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.017382    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.017475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.017396    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf8tl\" (UniqueName: \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-kube-api-access-wf8tl\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.017475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.017410    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.017475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.017424    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.017475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.017438    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.017475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.017451    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4d6af5-bc40-4313-b009-2c1d61ee9eab-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.047949 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.047921    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:49:48.085083 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.085060    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:49:48.117951 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.117929    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-token\" (UniqueName: \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-token\") pod \"90a417e0-9ddc-4187-aaf8-632e3e20401f\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") "
Oct 30 19:49:48.118082 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.117961    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-socket\") pod \"90a417e0-9ddc-4187-aaf8-632e3e20401f\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") "
Oct 30 19:49:48.118082 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.117983    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-data\") pod \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") "
Oct 30 19:49:48.118082 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118005    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-kube-api-access-wdz9f\" (UniqueName: \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-kube-api-access-wdz9f\") pod \"90a417e0-9ddc-4187-aaf8-632e3e20401f\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") "
Oct 30 19:49:48.118082 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118038    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-data\") pod \"90a417e0-9ddc-4187-aaf8-632e3e20401f\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") "
Oct 30 19:49:48.118286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118173    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-token\" (UniqueName: \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-token\") pod \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") "
Oct 30 19:49:48.118286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118209    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-envoy\") pod \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") "
Oct 30 19:49:48.118286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118248    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-kube-api-access-q5xxj\" (UniqueName: \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-kube-api-access-q5xxj\") pod \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") "
Oct 30 19:49:48.118286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118258    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-data" pod "3751f391-d63e-489b-9ae9-8ad55dd4f1b2" (UID: "3751f391-d63e-489b-9ae9-8ad55dd4f1b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.118496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118277    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-socket" pod "90a417e0-9ddc-4187-aaf8-632e3e20401f" (UID: "90a417e0-9ddc-4187-aaf8-632e3e20401f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.118496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118324    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-credential-socket\") pod \"90a417e0-9ddc-4187-aaf8-632e3e20401f\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") "
Oct 30 19:49:48.118496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118357    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-socket\") pod \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") "
Oct 30 19:49:48.118496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118387    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a417e0-9ddc-4187-aaf8-632e3e20401f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a417e0-9ddc-4187-aaf8-632e3e20401f-istiod-ca-cert\") pod \"90a417e0-9ddc-4187-aaf8-632e3e20401f\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") "
Oct 30 19:49:48.118496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118417    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-data" pod "90a417e0-9ddc-4187-aaf8-632e3e20401f" (UID: "90a417e0-9ddc-4187-aaf8-632e3e20401f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.118496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118428    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-envoy\") pod \"90a417e0-9ddc-4187-aaf8-632e3e20401f\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") "
Oct 30 19:49:48.118496 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118477    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-certs\") pod \"90a417e0-9ddc-4187-aaf8-632e3e20401f\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") "
Oct 30 19:49:48.121189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118531    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-podinfo\") pod \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") "
Oct 30 19:49:48.121189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118589    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istiod-ca-cert\") pod \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") "
Oct 30 19:49:48.121189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118618    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-podinfo\") pod \"90a417e0-9ddc-4187-aaf8-632e3e20401f\" (UID: \"90a417e0-9ddc-4187-aaf8-632e3e20401f\") "
Oct 30 19:49:48.121189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118632    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-socket" pod "3751f391-d63e-489b-9ae9-8ad55dd4f1b2" (UID: "3751f391-d63e-489b-9ae9-8ad55dd4f1b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.121189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118735    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-certs\") pod \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") "
Oct 30 19:49:48.121189 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.118864    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-credential-socket\") pod \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\" (UID: \"3751f391-d63e-489b-9ae9-8ad55dd4f1b2\") "
Oct 30 19:49:48.121482 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.119044    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-credential-socket" pod "90a417e0-9ddc-4187-aaf8-632e3e20401f" (UID: "90a417e0-9ddc-4187-aaf8-632e3e20401f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.121482 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.121467    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-certs" pod "3751f391-d63e-489b-9ae9-8ad55dd4f1b2" (UID: "3751f391-d63e-489b-9ae9-8ad55dd4f1b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.121787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.121763    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-credential-socket" pod "3751f391-d63e-489b-9ae9-8ad55dd4f1b2" (UID: "3751f391-d63e-489b-9ae9-8ad55dd4f1b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.122064 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.122012    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.122273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.122093    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.122273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.122111    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.122273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.122125    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.122273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.122140    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.122273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.122161    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.122273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.122175    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.122626 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.122606    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istiod-ca-cert" pod "3751f391-d63e-489b-9ae9-8ad55dd4f1b2" (UID: "3751f391-d63e-489b-9ae9-8ad55dd4f1b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:48.122708 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.122641    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a417e0-9ddc-4187-aaf8-632e3e20401f-istiod-ca-cert" pod "90a417e0-9ddc-4187-aaf8-632e3e20401f" (UID: "90a417e0-9ddc-4187-aaf8-632e3e20401f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:48.122903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.122875    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-certs" pod "90a417e0-9ddc-4187-aaf8-632e3e20401f" (UID: "90a417e0-9ddc-4187-aaf8-632e3e20401f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.125753 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.125720    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-podinfo" pod "3751f391-d63e-489b-9ae9-8ad55dd4f1b2" (UID: "3751f391-d63e-489b-9ae9-8ad55dd4f1b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:48.125984 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.125962    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-kube-api-access-q5xxj" pod "3751f391-d63e-489b-9ae9-8ad55dd4f1b2" (UID: "3751f391-d63e-489b-9ae9-8ad55dd4f1b2"). InnerVolumeSpecName "kube-api-access-q5xxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:48.126086 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.125998    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-kube-api-access-wdz9f" pod "90a417e0-9ddc-4187-aaf8-632e3e20401f" (UID: "90a417e0-9ddc-4187-aaf8-632e3e20401f"). InnerVolumeSpecName "kube-api-access-wdz9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:48.127182 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.127155    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-podinfo" pod "90a417e0-9ddc-4187-aaf8-632e3e20401f" (UID: "90a417e0-9ddc-4187-aaf8-632e3e20401f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:48.127283 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.127256    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-envoy" pod "90a417e0-9ddc-4187-aaf8-632e3e20401f" (UID: "90a417e0-9ddc-4187-aaf8-632e3e20401f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.127344 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.127302    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-token" pod "90a417e0-9ddc-4187-aaf8-632e3e20401f" (UID: "90a417e0-9ddc-4187-aaf8-632e3e20401f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:48.127344 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.127319    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-envoy" pod "3751f391-d63e-489b-9ae9-8ad55dd4f1b2" (UID: "3751f391-d63e-489b-9ae9-8ad55dd4f1b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.127413 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.127341    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-token" pod "3751f391-d63e-489b-9ae9-8ad55dd4f1b2" (UID: "3751f391-d63e-489b-9ae9-8ad55dd4f1b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:48.130024 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.130005    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:49:48.222618 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.222593    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-kube-api-access-5wf6f\" (UniqueName: \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-kube-api-access-5wf6f\") pod \"7c160df8-7210-45d5-b6bb-d42ae62c5310\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") "
Oct 30 19:49:48.223129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.222633    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-envoy\") pod \"7c160df8-7210-45d5-b6bb-d42ae62c5310\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") "
Oct 30 19:49:48.223129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.222664    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-socket\") pod \"7c160df8-7210-45d5-b6bb-d42ae62c5310\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") "
Oct 30 19:49:48.223129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.222699    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c160df8-7210-45d5-b6bb-d42ae62c5310-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c160df8-7210-45d5-b6bb-d42ae62c5310-istiod-ca-cert\") pod \"7c160df8-7210-45d5-b6bb-d42ae62c5310\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") "
Oct 30 19:49:48.223129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.222718    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-token\" (UniqueName: \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-token\") pod \"7c160df8-7210-45d5-b6bb-d42ae62c5310\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") "
Oct 30 19:49:48.223129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.222755    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-credential-socket\") pod \"7c160df8-7210-45d5-b6bb-d42ae62c5310\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") "
Oct 30 19:49:48.223129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.222912    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-podinfo\") pod \"7c160df8-7210-45d5-b6bb-d42ae62c5310\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") "
Oct 30 19:49:48.223129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.222949    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-certs\") pod \"7c160df8-7210-45d5-b6bb-d42ae62c5310\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") "
Oct 30 19:49:48.223129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223020    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-socket" pod "7c160df8-7210-45d5-b6bb-d42ae62c5310" (UID: "7c160df8-7210-45d5-b6bb-d42ae62c5310"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.223129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223106    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c160df8-7210-45d5-b6bb-d42ae62c5310-istiod-ca-cert" pod "7c160df8-7210-45d5-b6bb-d42ae62c5310" (UID: "7c160df8-7210-45d5-b6bb-d42ae62c5310"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:48.223129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223129    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-data\") pod \"7c160df8-7210-45d5-b6bb-d42ae62c5310\" (UID: \"7c160df8-7210-45d5-b6bb-d42ae62c5310\") "
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223215    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-credential-socket" pod "7c160df8-7210-45d5-b6bb-d42ae62c5310" (UID: "7c160df8-7210-45d5-b6bb-d42ae62c5310"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223436    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-certs" pod "7c160df8-7210-45d5-b6bb-d42ae62c5310" (UID: "7c160df8-7210-45d5-b6bb-d42ae62c5310"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223475    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-data" pod "7c160df8-7210-45d5-b6bb-d42ae62c5310" (UID: "7c160df8-7210-45d5-b6bb-d42ae62c5310"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223723    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223738    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a417e0-9ddc-4187-aaf8-632e3e20401f-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223747    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223755    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223769    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223777    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223785    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c160df8-7210-45d5-b6bb-d42ae62c5310-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223793    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223801    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223810    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdz9f\" (UniqueName: \"kubernetes.io/projected/90a417e0-9ddc-4187-aaf8-632e3e20401f-kube-api-access-wdz9f\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.223836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223838    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.224585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223851    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.224585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223864    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.224585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223877    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5xxj\" (UniqueName: \"kubernetes.io/projected/3751f391-d63e-489b-9ae9-8ad55dd4f1b2-kube-api-access-q5xxj\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.224585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223890    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.224585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.223898    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a417e0-9ddc-4187-aaf8-632e3e20401f-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.225539 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.225514    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-kube-api-access-5wf6f" pod "7c160df8-7210-45d5-b6bb-d42ae62c5310" (UID: "7c160df8-7210-45d5-b6bb-d42ae62c5310"). InnerVolumeSpecName "kube-api-access-5wf6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:48.225685 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.225667    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-envoy" pod "7c160df8-7210-45d5-b6bb-d42ae62c5310" (UID: "7c160df8-7210-45d5-b6bb-d42ae62c5310"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:48.225787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.225770    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-podinfo" pod "7c160df8-7210-45d5-b6bb-d42ae62c5310" (UID: "7c160df8-7210-45d5-b6bb-d42ae62c5310"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:48.225787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.225774    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-token" pod "7c160df8-7210-45d5-b6bb-d42ae62c5310" (UID: "7c160df8-7210-45d5-b6bb-d42ae62c5310"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:48.324783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.324655    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.324783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.324688    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wf6f\" (UniqueName: \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-kube-api-access-5wf6f\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.324783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.324700    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.324783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.324711    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c160df8-7210-45d5-b6bb-d42ae62c5310-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:48.764903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.764873    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"
Oct 30 19:49:48.765097 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.764877    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf" event={"ID":"3751f391-d63e-489b-9ae9-8ad55dd4f1b2","Type":"ContainerDied","Data":"1fcff1c8ed7894d19d4f98566cf583d8269a427f9b870fa9e4dde3e3a2d3df35"}
Oct 30 19:49:48.765097 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.765014    4420 scope.go:122] "RemoveContainer" containerID="f74af44b3e78ee7bab8d0e0be426f0bf4e76df8f6c6be2e238cff2fc86dfb9bc"
Oct 30 19:49:48.769891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.767591    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc" event={"ID":"7b4d6af5-bc40-4313-b009-2c1d61ee9eab","Type":"ContainerDied","Data":"e7e07435bb4879c135e2910a0a11626335408c19e5f7249ae2a82128c9739e59"}
Oct 30 19:49:48.769891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.767695    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"
Oct 30 19:49:48.771853 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.771836    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"
Oct 30 19:49:48.773461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.771910    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j" event={"ID":"90a417e0-9ddc-4187-aaf8-632e3e20401f","Type":"ContainerDied","Data":"de97e924432d865a1889a7fb141df7e97c36439c80f47dc9cf42f68a15813056"}
Oct 30 19:49:48.774553 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.774523    4420 generic.go:356] "Generic (PLEG): container finished" podID="7c160df8-7210-45d5-b6bb-d42ae62c5310" containerID="7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4" exitCode=0
Oct 30 19:49:48.774658 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.774572    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" event={"ID":"7c160df8-7210-45d5-b6bb-d42ae62c5310","Type":"ContainerDied","Data":"7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4"}
Oct 30 19:49:48.774658 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.774609    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc" event={"ID":"7c160df8-7210-45d5-b6bb-d42ae62c5310","Type":"ContainerDied","Data":"2b23a0146e8758c795537a68e0d44183e8d22801ffd4c0bcb82aa4f1b062723d"}
Oct 30 19:49:48.774767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.774674    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"
Oct 30 19:49:48.782360 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.782085    4420 scope.go:122] "RemoveContainer" containerID="613c61d3c390c4458a616688595ae8843331d58b2ad024f12f0fae0caae05605"
Oct 30 19:49:48.797115 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.797099    4420 scope.go:122] "RemoveContainer" containerID="761902ef60a86f2d4d82b8050ca9bfd3e6563820396f99491ceaa313bc728c9d"
Oct 30 19:49:48.861028 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.861005    4420 scope.go:122] "RemoveContainer" containerID="7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4"
Oct 30 19:49:48.907134 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.907049    4420 scope.go:122] "RemoveContainer" containerID="7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4"
Oct 30 19:49:48.909742 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:49:48.909666    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4\": container with ID starting with 7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4 not found: ID does not exist" containerID="7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4"
Oct 30 19:49:48.909742 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.909702    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4"} err="failed to get container status \"7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4\": rpc error: code = NotFound desc = could not find container \"7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4\": container with ID starting with 7fdfe9e55a25b1d6af0a107c1d293c2807ba24a9755c6ebded797bdec4061fe4 not found: ID does not exist"
Oct 30 19:49:48.913315 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.913292    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" path="/var/lib/kubelet/pods/8e0b55de-ce41-492a-846d-bd4533cfa6cc/volumes"
Oct 30 19:49:48.914223 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.914203    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" path="/var/lib/kubelet/pods/9813d487-1f24-47f9-9cea-47e54083d91e/volumes"
Oct 30 19:49:48.915035 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.915016    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" path="/var/lib/kubelet/pods/bfde16f9-2bea-4cf0-8c3f-7e689934de26/volumes"
Oct 30 19:49:48.936768 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.936745    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"]
Oct 30 19:49:48.987874 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:48.987849    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-dr8gc"]
Oct 30 19:49:49.047054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:49.046960    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"]
Oct 30 19:49:49.049936 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:49.049916    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-p2flf"]
Oct 30 19:49:49.298597 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:49.298500    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"]
Oct 30 19:49:49.299286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:49.299258    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-v8qbc"]
Oct 30 19:49:49.388330 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:49.388300    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"]
Oct 30 19:49:49.419842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:49.419792    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-szg2j"]
Oct 30 19:49:50.331982 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:50.324642    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"]
Oct 30 19:49:50.331982 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:50.325001    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy" containerID="cri-o://8d81ffe9d7805dd09ff5b9d8904ab26a8c57d84cc021992d26316978c9e62140" gracePeriod=30
Oct 30 19:49:50.398338 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:50.398305    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"]
Oct 30 19:49:50.419841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:50.413142    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy" containerID="cri-o://d0617c99a9206c5c539580b3a0870e597cb151dbdbaafdb59f92a1d6616c294c" gracePeriod=30
Oct 30 19:49:50.900520 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:50.900485    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" path="/var/lib/kubelet/pods/3751f391-d63e-489b-9ae9-8ad55dd4f1b2/volumes"
Oct 30 19:49:50.901025 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:50.901009    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" path="/var/lib/kubelet/pods/7b4d6af5-bc40-4313-b009-2c1d61ee9eab/volumes"
Oct 30 19:49:50.901432 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:50.901417    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c160df8-7210-45d5-b6bb-d42ae62c5310" path="/var/lib/kubelet/pods/7c160df8-7210-45d5-b6bb-d42ae62c5310/volumes"
Oct 30 19:49:50.902208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:50.902189    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" path="/var/lib/kubelet/pods/90a417e0-9ddc-4187-aaf8-632e3e20401f/volumes"
Oct 30 19:49:52.042841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.042542    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"]
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043527    4420 cpu_manager.go:414] "containerMap: removing container" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043552    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043587    4420 cpu_manager.go:414] "containerMap: removing container" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043595    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043609    4420 cpu_manager.go:414] "containerMap: removing container" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043618    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043631    4420 cpu_manager.go:414] "containerMap: removing container" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043638    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043653    4420 cpu_manager.go:414] "containerMap: removing container" podUID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043661    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043676    4420 cpu_manager.go:414] "containerMap: removing container" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043683    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043711    4420 cpu_manager.go:414] "containerMap: removing container" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043717    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043739    4420 cpu_manager.go:414] "containerMap: removing container" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043745    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043755    4420 cpu_manager.go:414] "containerMap: removing container" podUID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043762    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043782    4420 cpu_manager.go:414] "containerMap: removing container" podUID="7c160df8-7210-45d5-b6bb-d42ae62c5310" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043789    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c160df8-7210-45d5-b6bb-d42ae62c5310" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043950    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043968    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043977    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9813d487-1f24-47f9-9cea-47e54083d91e" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.043989    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0b55de-ce41-492a-846d-bd4533cfa6cc" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.044000    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4d6af5-bc40-4313-b009-2c1d61ee9eab" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.044008    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0379827-bfec-4f2c-86b0-fb38be8d6281" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.044022    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfde16f9-2bea-4cf0-8c3f-7e689934de26" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.044030    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3751f391-d63e-489b-9ae9-8ad55dd4f1b2" containerName="istio-proxy"
Oct 30 19:49:52.056881 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.044040    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c160df8-7210-45d5-b6bb-d42ae62c5310" containerName="istio-proxy"
Oct 30 19:49:52.088256 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.088225    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.104542 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.104394    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"]
Oct 30 19:49:52.143809 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.143761    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-86e4a7-openshift-default-dockercfg-mt9h7\""
Oct 30 19:49:52.155894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.155579    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-data\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.155894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.155669    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-credential-socket\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.155894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.155742    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-certs\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.155894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.155788    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-socket\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.156450 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.156348    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-envoy\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.161202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.161157    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts42p\" (UniqueName: \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-kube-api-access-ts42p\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.161504 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.161261    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-token\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.166731 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.161545    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac27ddeb-8cb2-473a-b036-3f0201411622-istiod-ca-cert\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.166731 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.161591    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-podinfo\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.262898 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.262867    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-data\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263076 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.262933    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-credential-socket\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263076 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.262993    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-certs\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263076 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.263036    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-socket\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263248 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.263232    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-envoy\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.263372    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts42p\" (UniqueName: \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-kube-api-access-ts42p\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.263387    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-credential-socket\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.263430    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-data\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.263453    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-token\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.263501    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac27ddeb-8cb2-473a-b036-3f0201411622-istiod-ca-cert\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.263527    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-certs\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.263544    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-podinfo\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.263615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.263571    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-socket\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.264438 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.264316    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac27ddeb-8cb2-473a-b036-3f0201411622-istiod-ca-cert\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.265937 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.265912    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-envoy\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.266674 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.266653    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-podinfo\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.320715 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.320639    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-token\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.320859 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.320711    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts42p\" (UniqueName: \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-kube-api-access-ts42p\") pod \"e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") " pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.431583 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.431552    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:52.865762 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.865729    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"]
Oct 30 19:49:52.871970 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.871937    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:49:52.872072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.872024    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:49:52.872072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:52.872066    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:49:53.651992 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.651949    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"]
Oct 30 19:49:53.652647 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.652620    4420 cpu_manager.go:414] "containerMap: removing container" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy"
Oct 30 19:49:53.652647 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.652636    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy"
Oct 30 19:49:53.652878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.652780    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b0996b-c55b-4e37-8220-1302701f6260" containerName="istio-proxy"
Oct 30 19:49:53.652878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.652791    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a417e0-9ddc-4187-aaf8-632e3e20401f" containerName="istio-proxy"
Oct 30 19:49:53.657782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.657760    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.711836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.711051    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cb14ed-openshift-default-dockercfg-f4qhr\""
Oct 30 19:49:53.764286 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.764251    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"]
Oct 30 19:49:53.787742 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.787702    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-socket\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.787929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.787768    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-token\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.787995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.787928    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74aa3ef-91cc-4ece-b74e-c3332572e110-istiod-ca-cert\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.788067 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.788000    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq69q\" (UniqueName: \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-kube-api-access-xq69q\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.788209 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.788186    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-podinfo\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.788279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.788230    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-data\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.788349 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.788328    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-envoy\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.792216 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.788395    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-certs\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.792216 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.789006    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-credential-socket\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.879637 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.879599    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" event={"ID":"ac27ddeb-8cb2-473a-b036-3f0201411622","Type":"ContainerStarted","Data":"a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1"}
Oct 30 19:49:53.879801 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.879681    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" event={"ID":"ac27ddeb-8cb2-473a-b036-3f0201411622","Type":"ContainerStarted","Data":"cbbc60820adb9e99b22ddca48023e2ebf2ccbec4488f4334d61a462c14d8dfa0"}
Oct 30 19:49:53.893705 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.893679    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-socket\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.893864 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.893715    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-token\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.893864 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.893761    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74aa3ef-91cc-4ece-b74e-c3332572e110-istiod-ca-cert\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.893864 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.893795    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq69q\" (UniqueName: \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-kube-api-access-xq69q\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.894015 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.893874    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-podinfo\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.894015 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.893902    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-data\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.894136 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.894068    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-envoy\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.894136 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.894113    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-certs\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.894246 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.894177    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-credential-socket\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.894246 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.894224    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-data\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.894361 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.894115    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-socket\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.894443 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.894421    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-certs\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.894510 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.894468    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-credential-socket\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.894638 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.894619    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74aa3ef-91cc-4ece-b74e-c3332572e110-istiod-ca-cert\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.896604 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.896583    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-envoy\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.896982 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.896961    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-podinfo\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.950618 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.950549    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-token\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.951253 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.951227    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq69q\" (UniqueName: \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-kube-api-access-xq69q\") pod \"e2e-api-a-cb14ed-openshift-default-68f97495c9-68895\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") " pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:53.972730 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:53.972708    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:54.000542 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.000498    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" podStartSLOduration=3.000480418 podStartE2EDuration="3.000480418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:53.999173471 +0000 UTC m=+2041.210390616" watchObservedRunningTime="2026-10-30 19:49:54.000480418 +0000 UTC m=+2041.211697563"
Oct 30 19:49:54.413012 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.412985    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"]
Oct 30 19:49:54.413688 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:49:54.413654    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74aa3ef_91cc_4ece_b74e_c3332572e110.slice/crio-2c96c7dc4e949c14d2b0b0f6cc50fbdb5ce71b34f47ebd9b51000e698032eec9 WatchSource:0}: Error finding container 2c96c7dc4e949c14d2b0b0f6cc50fbdb5ce71b34f47ebd9b51000e698032eec9: Status 404 returned error can't find the container with id 2c96c7dc4e949c14d2b0b0f6cc50fbdb5ce71b34f47ebd9b51000e698032eec9
Oct 30 19:49:54.417987 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.417946    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:49:54.418093 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.418036    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:49:54.418152 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.418096    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:49:54.432764 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.432686    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:54.458952 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.458925    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:54.888092 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.888061    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" event={"ID":"b74aa3ef-91cc-4ece-b74e-c3332572e110","Type":"ContainerStarted","Data":"502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19"}
Oct 30 19:49:54.888551 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.888122    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" event={"ID":"b74aa3ef-91cc-4ece-b74e-c3332572e110","Type":"ContainerStarted","Data":"2c96c7dc4e949c14d2b0b0f6cc50fbdb5ce71b34f47ebd9b51000e698032eec9"}
Oct 30 19:49:54.898093 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.898071    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:54.898252 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.898128    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:49:54.973614 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:54.973588    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:55.007166 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:55.007119    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" podStartSLOduration=2.007104112 podStartE2EDuration="2.007104112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:55.00327588 +0000 UTC m=+2042.214493029" watchObservedRunningTime="2026-10-30 19:49:55.007104112 +0000 UTC m=+2042.218321257"
Oct 30 19:49:55.918927 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:55.906891    4420 generic.go:356] "Generic (PLEG): container finished" podID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerID="8d81ffe9d7805dd09ff5b9d8904ab26a8c57d84cc021992d26316978c9e62140" exitCode=0
Oct 30 19:49:55.918927 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:55.907052    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" event={"ID":"099d010e-c8ad-40cc-ac3b-f445bb1b52fa","Type":"ContainerDied","Data":"8d81ffe9d7805dd09ff5b9d8904ab26a8c57d84cc021992d26316978c9e62140"}
Oct 30 19:49:55.918927 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:55.908945    4420 generic.go:356] "Generic (PLEG): container finished" podID="8c49526a-3a13-4449-9658-b7a42840139e" containerID="d0617c99a9206c5c539580b3a0870e597cb151dbdbaafdb59f92a1d6616c294c" exitCode=0
Oct 30 19:49:55.918927 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:55.909864    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" event={"ID":"8c49526a-3a13-4449-9658-b7a42840139e","Type":"ContainerDied","Data":"d0617c99a9206c5c539580b3a0870e597cb151dbdbaafdb59f92a1d6616c294c"}
Oct 30 19:49:55.971628 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:55.971598    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:49:55.980315 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:55.980289    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:55.997785 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:55.997765    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:49:56.132446 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132416    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-certs\") pod \"8c49526a-3a13-4449-9658-b7a42840139e\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") "
Oct 30 19:49:56.132632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132455    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c49526a-3a13-4449-9658-b7a42840139e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c49526a-3a13-4449-9658-b7a42840139e-istio-podinfo\") pod \"8c49526a-3a13-4449-9658-b7a42840139e\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") "
Oct 30 19:49:56.132632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132522    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-data\") pod \"8c49526a-3a13-4449-9658-b7a42840139e\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") "
Oct 30 19:49:56.132632 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132565    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-data\") pod \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") "
Oct 30 19:49:56.132789 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132664    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-kube-api-access-b9h6w\" (UniqueName: \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-kube-api-access-b9h6w\") pod \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") "
Oct 30 19:49:56.132789 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132698    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-socket\") pod \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") "
Oct 30 19:49:56.132789 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132747    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-envoy\") pod \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") "
Oct 30 19:49:56.132789 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132747    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-certs" pod "8c49526a-3a13-4449-9658-b7a42840139e" (UID: "8c49526a-3a13-4449-9658-b7a42840139e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.132789 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132780    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c49526a-3a13-4449-9658-b7a42840139e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c49526a-3a13-4449-9658-b7a42840139e-istiod-ca-cert\") pod \"8c49526a-3a13-4449-9658-b7a42840139e\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") "
Oct 30 19:49:56.133058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132800    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-data" pod "8c49526a-3a13-4449-9658-b7a42840139e" (UID: "8c49526a-3a13-4449-9658-b7a42840139e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.133058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132856    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-token\" (UniqueName: \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-token\") pod \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") "
Oct 30 19:49:56.133058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132910    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-credential-socket\") pod \"8c49526a-3a13-4449-9658-b7a42840139e\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") "
Oct 30 19:49:56.133058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132942    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-podinfo\") pod \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") "
Oct 30 19:49:56.133058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132949    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-data" pod "099d010e-c8ad-40cc-ac3b-f445bb1b52fa" (UID: "099d010e-c8ad-40cc-ac3b-f445bb1b52fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.133058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.132969    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istiod-ca-cert\") pod \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") "
Oct 30 19:49:56.133058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.133024    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-socket" pod "099d010e-c8ad-40cc-ac3b-f445bb1b52fa" (UID: "099d010e-c8ad-40cc-ac3b-f445bb1b52fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.133058 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.133039    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-kube-api-access-ncsh4\" (UniqueName: \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-kube-api-access-ncsh4\") pod \"8c49526a-3a13-4449-9658-b7a42840139e\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") "
Oct 30 19:49:56.133439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.133072    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-certs\") pod \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") "
Oct 30 19:49:56.133439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.133135    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-istio-token\" (UniqueName: \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-istio-token\") pod \"8c49526a-3a13-4449-9658-b7a42840139e\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") "
Oct 30 19:49:56.133439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.133173    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-credential-socket\") pod \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\" (UID: \"099d010e-c8ad-40cc-ac3b-f445bb1b52fa\") "
Oct 30 19:49:56.133439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.133214    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-envoy\") pod \"8c49526a-3a13-4449-9658-b7a42840139e\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") "
Oct 30 19:49:56.133439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.133249    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istiod-ca-cert" pod "099d010e-c8ad-40cc-ac3b-f445bb1b52fa" (UID: "099d010e-c8ad-40cc-ac3b-f445bb1b52fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:56.133439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.133281    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-socket\") pod \"8c49526a-3a13-4449-9658-b7a42840139e\" (UID: \"8c49526a-3a13-4449-9658-b7a42840139e\") "
Oct 30 19:49:56.133903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.133529    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-credential-socket" pod "8c49526a-3a13-4449-9658-b7a42840139e" (UID: "8c49526a-3a13-4449-9658-b7a42840139e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.133903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.133595    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c49526a-3a13-4449-9658-b7a42840139e-istiod-ca-cert" pod "8c49526a-3a13-4449-9658-b7a42840139e" (UID: "8c49526a-3a13-4449-9658-b7a42840139e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:49:56.134346 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.134292    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.134346 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.134318    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.134346 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.134331    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.134346 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.134344    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.134591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.134359    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c49526a-3a13-4449-9658-b7a42840139e-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.134591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.134373    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.134591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.134379    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-socket" pod "8c49526a-3a13-4449-9658-b7a42840139e" (UID: "8c49526a-3a13-4449-9658-b7a42840139e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.134591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.134388    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.134591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.134455    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-credential-socket" pod "099d010e-c8ad-40cc-ac3b-f445bb1b52fa" (UID: "099d010e-c8ad-40cc-ac3b-f445bb1b52fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.134929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.134893    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-certs" pod "099d010e-c8ad-40cc-ac3b-f445bb1b52fa" (UID: "099d010e-c8ad-40cc-ac3b-f445bb1b52fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.136044 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.136015    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-token" pod "099d010e-c8ad-40cc-ac3b-f445bb1b52fa" (UID: "099d010e-c8ad-40cc-ac3b-f445bb1b52fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:56.136136 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.136096    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c49526a-3a13-4449-9658-b7a42840139e-istio-podinfo" pod "8c49526a-3a13-4449-9658-b7a42840139e" (UID: "8c49526a-3a13-4449-9658-b7a42840139e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:56.136208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.136179    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-istio-token" pod "8c49526a-3a13-4449-9658-b7a42840139e" (UID: "8c49526a-3a13-4449-9658-b7a42840139e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:56.136333 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.136223    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-envoy" pod "099d010e-c8ad-40cc-ac3b-f445bb1b52fa" (UID: "099d010e-c8ad-40cc-ac3b-f445bb1b52fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.136404 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.136388    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-podinfo" pod "099d010e-c8ad-40cc-ac3b-f445bb1b52fa" (UID: "099d010e-c8ad-40cc-ac3b-f445bb1b52fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:49:56.136970 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.136936    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-envoy" pod "8c49526a-3a13-4449-9658-b7a42840139e" (UID: "8c49526a-3a13-4449-9658-b7a42840139e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:49:56.137300 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.137272    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-kube-api-access-b9h6w" pod "099d010e-c8ad-40cc-ac3b-f445bb1b52fa" (UID: "099d010e-c8ad-40cc-ac3b-f445bb1b52fa"). InnerVolumeSpecName "kube-api-access-b9h6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:56.137378 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.137312    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-kube-api-access-ncsh4" pod "8c49526a-3a13-4449-9658-b7a42840139e" (UID: "8c49526a-3a13-4449-9658-b7a42840139e"). InnerVolumeSpecName "kube-api-access-ncsh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:49:56.235615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.235510    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.235615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.235534    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.235615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.235542    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.235615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.235552    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncsh4\" (UniqueName: \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-kube-api-access-ncsh4\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.235615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.235560    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.235615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.235568    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c49526a-3a13-4449-9658-b7a42840139e-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.235615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.235577    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.235615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.235584    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.235615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.235592    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c49526a-3a13-4449-9658-b7a42840139e-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.235615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.235601    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c49526a-3a13-4449-9658-b7a42840139e-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.235615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.235610    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9h6w\" (UniqueName: \"kubernetes.io/projected/099d010e-c8ad-40cc-ac3b-f445bb1b52fa-kube-api-access-b9h6w\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:49:56.916763 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.916736    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"
Oct 30 19:49:56.916763 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.916753    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww" event={"ID":"099d010e-c8ad-40cc-ac3b-f445bb1b52fa","Type":"ContainerDied","Data":"3b33835b2524c38946de1b57a881c95107d16c6e2f1c55057b97a0ea920a30eb"}
Oct 30 19:49:56.916941 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.916807    4420 scope.go:122] "RemoveContainer" containerID="8d81ffe9d7805dd09ff5b9d8904ab26a8c57d84cc021992d26316978c9e62140"
Oct 30 19:49:56.918388 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.918314    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"
Oct 30 19:49:56.918388 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.918328    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg" event={"ID":"8c49526a-3a13-4449-9658-b7a42840139e","Type":"ContainerDied","Data":"93e912ceb3fea537238d22ca021ec4681c60cb385ff2e74e2541160f47590804"}
Oct 30 19:49:56.918771 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.918701    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:56.920335 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.920316    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:49:56.927378 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:56.927356    4420 scope.go:122] "RemoveContainer" containerID="d0617c99a9206c5c539580b3a0870e597cb151dbdbaafdb59f92a1d6616c294c"
Oct 30 19:49:57.202415 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:57.202336    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"]
Oct 30 19:49:57.206030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:57.206007    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-lbvww"]
Oct 30 19:49:57.311380 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:57.311354    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"]
Oct 30 19:49:57.317229 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:57.317205    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-8lwmg"]
Oct 30 19:49:58.897239 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:58.897208    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" path="/var/lib/kubelet/pods/099d010e-c8ad-40cc-ac3b-f445bb1b52fa/volumes"
Oct 30 19:49:58.897669 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:49:58.897655    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c49526a-3a13-4449-9658-b7a42840139e" path="/var/lib/kubelet/pods/8c49526a-3a13-4449-9658-b7a42840139e/volumes"
Oct 30 19:50:04.850489 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:04.850458    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"]
Oct 30 19:50:04.850897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:04.850686    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy" containerID="cri-o://32c36ecacbd0f05b2bd5c16c1e995f508c9213c5b850efb525bae65ba45790fe" gracePeriod=30
Oct 30 19:50:05.197635 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:50:05.197597    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:50:10.009266 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.009232    4420 generic.go:356] "Generic (PLEG): container finished" podID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerID="32c36ecacbd0f05b2bd5c16c1e995f508c9213c5b850efb525bae65ba45790fe" exitCode=0
Oct 30 19:50:10.009619 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.009391    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" event={"ID":"7829d2b9-e06d-41c6-98a7-01c17d9b29c8","Type":"ContainerDied","Data":"32c36ecacbd0f05b2bd5c16c1e995f508c9213c5b850efb525bae65ba45790fe"}
Oct 30 19:50:10.009619 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.009473    4420 scope.go:122] "RemoveContainer" containerID="099516948db982bb35c82430137ae1c3071fd897659898a3cf5bfdc48c1e3568"
Oct 30 19:50:10.122156 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.122097    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:50:10.242712 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.242681    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-credential-socket\") pod \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") "
Oct 30 19:50:10.242907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.242719    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-data\") pod \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") "
Oct 30 19:50:10.242907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.242784    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-socket\") pod \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") "
Oct 30 19:50:10.242907 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.242865    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-envoy\") pod \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") "
Oct 30 19:50:10.243054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.242907    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-kube-api-access-bv9bl\" (UniqueName: \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-kube-api-access-bv9bl\") pod \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") "
Oct 30 19:50:10.243054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.242944    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-certs\") pod \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") "
Oct 30 19:50:10.243054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.242984    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-token\" (UniqueName: \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-token\") pod \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") "
Oct 30 19:50:10.243054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.242985    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-credential-socket" pod "7829d2b9-e06d-41c6-98a7-01c17d9b29c8" (UID: "7829d2b9-e06d-41c6-98a7-01c17d9b29c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:50:10.243212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.243073    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-data" pod "7829d2b9-e06d-41c6-98a7-01c17d9b29c8" (UID: "7829d2b9-e06d-41c6-98a7-01c17d9b29c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:50:10.243212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.243080    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-podinfo\") pod \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") "
Oct 30 19:50:10.243212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.243142    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-socket" pod "7829d2b9-e06d-41c6-98a7-01c17d9b29c8" (UID: "7829d2b9-e06d-41c6-98a7-01c17d9b29c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:50:10.243321 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.243219    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-certs" pod "7829d2b9-e06d-41c6-98a7-01c17d9b29c8" (UID: "7829d2b9-e06d-41c6-98a7-01c17d9b29c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:50:10.243321 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.243276    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istiod-ca-cert\") pod \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\" (UID: \"7829d2b9-e06d-41c6-98a7-01c17d9b29c8\") "
Oct 30 19:50:10.243865 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.243803    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istiod-ca-cert" pod "7829d2b9-e06d-41c6-98a7-01c17d9b29c8" (UID: "7829d2b9-e06d-41c6-98a7-01c17d9b29c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:50:10.243982 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.243961    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:10.244045 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.243983    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:10.244045 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.243997    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:10.244045 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.244011    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:10.245329 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.245302    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-podinfo" pod "7829d2b9-e06d-41c6-98a7-01c17d9b29c8" (UID: "7829d2b9-e06d-41c6-98a7-01c17d9b29c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:50:10.245936 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.245918    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-envoy" pod "7829d2b9-e06d-41c6-98a7-01c17d9b29c8" (UID: "7829d2b9-e06d-41c6-98a7-01c17d9b29c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:50:10.247144 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.247115    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-kube-api-access-bv9bl" pod "7829d2b9-e06d-41c6-98a7-01c17d9b29c8" (UID: "7829d2b9-e06d-41c6-98a7-01c17d9b29c8"). InnerVolumeSpecName "kube-api-access-bv9bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:10.247306 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.247287    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-token" pod "7829d2b9-e06d-41c6-98a7-01c17d9b29c8" (UID: "7829d2b9-e06d-41c6-98a7-01c17d9b29c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:50:10.345078 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.345052    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:10.345078 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.345077    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:10.345225 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.345087    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:10.345225 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.345095    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:10.345225 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:10.345104    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv9bl\" (UniqueName: \"kubernetes.io/projected/7829d2b9-e06d-41c6-98a7-01c17d9b29c8-kube-api-access-bv9bl\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:50:11.017241 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:11.017215    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"
Oct 30 19:50:11.017685 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:11.017254    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz" event={"ID":"7829d2b9-e06d-41c6-98a7-01c17d9b29c8","Type":"ContainerDied","Data":"fb805e5824a4e873e3929c1403cfec2dc7211d3a87c05b82fc70c3a5c71069ff"}
Oct 30 19:50:11.017685 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:11.017321    4420 scope.go:122] "RemoveContainer" containerID="32c36ecacbd0f05b2bd5c16c1e995f508c9213c5b850efb525bae65ba45790fe"
Oct 30 19:50:11.181704 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:11.181623    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"]
Oct 30 19:50:11.230725 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:11.230699    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-df789477-openshift-default-5f45898cd9-wmssz"]
Oct 30 19:50:12.901027 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:12.900987    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" path="/var/lib/kubelet/pods/7829d2b9-e06d-41c6-98a7-01c17d9b29c8/volumes"
Oct 30 19:50:16.484806 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.484773    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"]
Oct 30 19:50:16.485551 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.485508    4420 cpu_manager.go:414] "containerMap: removing container" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy"
Oct 30 19:50:16.485551 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.485532    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy"
Oct 30 19:50:16.485551 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.485554    4420 cpu_manager.go:414] "containerMap: removing container" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy"
Oct 30 19:50:16.485791 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.485563    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy"
Oct 30 19:50:16.485791 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.485595    4420 cpu_manager.go:414] "containerMap: removing container" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy"
Oct 30 19:50:16.485791 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.485603    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy"
Oct 30 19:50:16.485791 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.485635    4420 cpu_manager.go:414] "containerMap: removing container" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy"
Oct 30 19:50:16.485791 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.485642    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy"
Oct 30 19:50:16.485791 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.485748    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy"
Oct 30 19:50:16.485791 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.485761    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c49526a-3a13-4449-9658-b7a42840139e" containerName="istio-proxy"
Oct 30 19:50:16.485791 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.485767    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="099d010e-c8ad-40cc-ac3b-f445bb1b52fa" containerName="istio-proxy"
Oct 30 19:50:16.489069 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.489054    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.544004 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.543959    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"]
Oct 30 19:50:16.604585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.604555    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-daa1ac-openshift-default-dockercfg-qh5md\""
Oct 30 19:50:16.621293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.621264    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415f5443-053d-4e88-814d-91952beb32f5-istio-podinfo\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.621402 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.621301    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-envoy\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.621448 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.621406    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldqmp\" (UniqueName: \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-kube-api-access-ldqmp\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.621518 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.621495    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-istio-token\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.621555 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.621542    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-credential-socket\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.621603 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.621565    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-socket\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.621685 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.621652    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-data\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.621806 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.621780    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-certs\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.621914 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.621895    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415f5443-053d-4e88-814d-91952beb32f5-istiod-ca-cert\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.724747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.724708    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-socket\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.735437 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.735347    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-data\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.735625 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.735473    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-certs\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.735625 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.735558    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415f5443-053d-4e88-814d-91952beb32f5-istiod-ca-cert\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.735900 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.735680    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415f5443-053d-4e88-814d-91952beb32f5-istio-podinfo\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.735900 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.735728    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-envoy\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.735900 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.735877    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldqmp\" (UniqueName: \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-kube-api-access-ldqmp\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.736120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.735937    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-istio-token\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.736120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.735991    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-credential-socket\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.740846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.736396    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-credential-socket\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.740846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.726965    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-socket\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.740846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.736649    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-data\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.740846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.736908    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-certs\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.740846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.737532    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415f5443-053d-4e88-814d-91952beb32f5-istiod-ca-cert\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.756775 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.756745    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-envoy\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.765246 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.765215    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415f5443-053d-4e88-814d-91952beb32f5-istio-podinfo\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.811119 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.811088    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-istio-token\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:16.811302 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:16.811161    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldqmp\" (UniqueName: \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-kube-api-access-ldqmp\") pod \"e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") " pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:17.104267 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:17.104194    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:17.535510 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:17.535473    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"]
Oct 30 19:50:17.536687 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:50:17.536663    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415f5443_053d_4e88_814d_91952beb32f5.slice/crio-47faa47b4b910856bf5f1f3740428ea22d1a431ffe5fa55fef5b823959499cfd WatchSource:0}: Error finding container 47faa47b4b910856bf5f1f3740428ea22d1a431ffe5fa55fef5b823959499cfd: Status 404 returned error can't find the container with id 47faa47b4b910856bf5f1f3740428ea22d1a431ffe5fa55fef5b823959499cfd
Oct 30 19:50:17.539337 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:17.539304    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:17.539439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:17.539373    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:17.539439 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:17.539413    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:18.063921 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:18.063878    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" event={"ID":"415f5443-053d-4e88-814d-91952beb32f5","Type":"ContainerStarted","Data":"af82ba18af013c0ce5d8a6004368c71f9556a32f48e2ec64559596ae2b03d4f6"}
Oct 30 19:50:18.064092 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:18.063939    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" event={"ID":"415f5443-053d-4e88-814d-91952beb32f5","Type":"ContainerStarted","Data":"47faa47b4b910856bf5f1f3740428ea22d1a431ffe5fa55fef5b823959499cfd"}
Oct 30 19:50:18.104668 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:18.104644    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:18.110550 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:18.110532    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:18.278673 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:18.278623    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" podStartSLOduration=2.2786096750000002 podStartE2EDuration="2.278609675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:18.228337708 +0000 UTC m=+2065.439554863" watchObservedRunningTime="2026-10-30 19:50:18.278609675 +0000 UTC m=+2065.489826820"
Oct 30 19:50:19.069607 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:19.069570    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:19.070404 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:19.070388    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:50:24.900793 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:24.900754    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"]
Oct 30 19:50:24.901744 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:24.901721    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7829d2b9-e06d-41c6-98a7-01c17d9b29c8" containerName="istio-proxy"
Oct 30 19:50:24.905474 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:24.905453    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:50:24.962152 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:24.962119    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"]
Oct 30 19:50:25.006712 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:25.006681    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-daa1ac\""
Oct 30 19:50:25.006953 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:25.006703    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-daa1ac-dockercfg-2l242\""
Oct 30 19:50:25.062211 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:25.062175    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ab0175-196a-4ea7-be0a-69d8973befd2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp\" (UID: \"39ab0175-196a-4ea7-be0a-69d8973befd2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:50:25.062417 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:25.062370    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-725k4\" (UniqueName: \"kubernetes.io/projected/39ab0175-196a-4ea7-be0a-69d8973befd2-kube-api-access-725k4\") pod \"payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp\" (UID: \"39ab0175-196a-4ea7-be0a-69d8973befd2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:50:25.163414 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:25.163341    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ab0175-196a-4ea7-be0a-69d8973befd2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp\" (UID: \"39ab0175-196a-4ea7-be0a-69d8973befd2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:50:25.163574 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:25.163436    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-725k4\" (UniqueName: \"kubernetes.io/projected/39ab0175-196a-4ea7-be0a-69d8973befd2-kube-api-access-725k4\") pod \"payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp\" (UID: \"39ab0175-196a-4ea7-be0a-69d8973befd2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:50:25.164322 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:25.164285    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ab0175-196a-4ea7-be0a-69d8973befd2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp\" (UID: \"39ab0175-196a-4ea7-be0a-69d8973befd2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:50:25.220373 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:25.220349    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-725k4\" (UniqueName: \"kubernetes.io/projected/39ab0175-196a-4ea7-be0a-69d8973befd2-kube-api-access-725k4\") pod \"payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp\" (UID: \"39ab0175-196a-4ea7-be0a-69d8973befd2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:50:25.220513 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:25.220498    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:50:25.598675 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:25.598651    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"]
Oct 30 19:50:25.600784 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:50:25.600756    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ab0175_196a_4ea7_be0a_69d8973befd2.slice/crio-6c61abd84ae3ce62e45995246e954fc1730ab2d942ced72e9f2ddf5cef7b84fc WatchSource:0}: Error finding container 6c61abd84ae3ce62e45995246e954fc1730ab2d942ced72e9f2ddf5cef7b84fc: Status 404 returned error can't find the container with id 6c61abd84ae3ce62e45995246e954fc1730ab2d942ced72e9f2ddf5cef7b84fc
Oct 30 19:50:26.112976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:26.112941    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp" event={"ID":"39ab0175-196a-4ea7-be0a-69d8973befd2","Type":"ContainerStarted","Data":"3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89"}
Oct 30 19:50:26.113417 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:26.113005    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp" event={"ID":"39ab0175-196a-4ea7-be0a-69d8973befd2","Type":"ContainerStarted","Data":"6c61abd84ae3ce62e45995246e954fc1730ab2d942ced72e9f2ddf5cef7b84fc"}
Oct 30 19:50:26.113417 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:26.113080    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:50:26.230170 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:26.230107    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp" podStartSLOduration=2.230088969 podStartE2EDuration="2.230088969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:26.225381249 +0000 UTC m=+2073.436598386" watchObservedRunningTime="2026-10-30 19:50:26.230088969 +0000 UTC m=+2073.441306116"
Oct 30 19:50:35.195360 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:50:35.195322    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:50:37.122248 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:37.122220    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:50:40.452368 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.452334    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"]
Oct 30 19:50:40.457950 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.457929    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.524320 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.524288    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6873e0-openshift-default-dockercfg-gfxk6\""
Oct 30 19:50:40.586700 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.586669    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-data\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.587186 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.587127    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-credential-socket\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.588566 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.588542    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-socket\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.588694 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.588617    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-certs\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.588694 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.588655    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-podinfo\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.588694 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.588683    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-envoy\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.588910 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.588711    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5bx8\" (UniqueName: \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-kube-api-access-k5bx8\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.588910 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.588759    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e17a94-a1f3-4273-a8f5-458b8d478a16-istiod-ca-cert\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.588910 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.588804    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-token\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.607623 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.607589    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"]
Oct 30 19:50:40.689903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.689871    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-credential-socket\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.690208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.689937    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-socket\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.690208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.689986    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-certs\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.690208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.690011    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-podinfo\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.690208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.690162    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-envoy\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.690208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.690213    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5bx8\" (UniqueName: \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-kube-api-access-k5bx8\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.690537 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.690261    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e17a94-a1f3-4273-a8f5-458b8d478a16-istiod-ca-cert\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.690537 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.690309    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-token\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.690537 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.690337    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-credential-socket\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.690537 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.690458    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-data\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.691113 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.691087    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e17a94-a1f3-4273-a8f5-458b8d478a16-istiod-ca-cert\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.691113 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.691103    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-certs\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.691306 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.691282    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-socket\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.691387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.691364    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-data\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.693212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.693187    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-podinfo\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.693402 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.693383    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-envoy\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.747134 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.747063    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-token\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.747371 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.747349    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5bx8\" (UniqueName: \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-kube-api-access-k5bx8\") pod \"e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") " pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:40.771888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:40.771860    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:41.209803 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:41.209775    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"]
Oct 30 19:50:41.210528 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:50:41.210501    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e17a94_a1f3_4273_a8f5_458b8d478a16.slice/crio-4b7c4de992045661ac065d323d6b01a7ec5d2fc960359bbe7266d16b2838bc08 WatchSource:0}: Error finding container 4b7c4de992045661ac065d323d6b01a7ec5d2fc960359bbe7266d16b2838bc08: Status 404 returned error can't find the container with id 4b7c4de992045661ac065d323d6b01a7ec5d2fc960359bbe7266d16b2838bc08
Oct 30 19:50:41.213232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:41.213205    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:41.213299 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:41.213266    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:41.213299 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:41.213294    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:50:42.217285 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:42.217254    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" event={"ID":"72e17a94-a1f3-4273-a8f5-458b8d478a16","Type":"ContainerStarted","Data":"714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553"}
Oct 30 19:50:42.217691 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:42.217304    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" event={"ID":"72e17a94-a1f3-4273-a8f5-458b8d478a16","Type":"ContainerStarted","Data":"4b7c4de992045661ac065d323d6b01a7ec5d2fc960359bbe7266d16b2838bc08"}
Oct 30 19:50:42.337385 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:42.337318    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" podStartSLOduration=2.337303331 podStartE2EDuration="2.337303331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:42.334352058 +0000 UTC m=+2089.545569203" watchObservedRunningTime="2026-10-30 19:50:42.337303331 +0000 UTC m=+2089.548520475"
Oct 30 19:50:42.772980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:42.772947    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:42.778410 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:42.778386    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:43.223879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:43.223848    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:50:43.224879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:50:43.224858    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:51:05.197755 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:51:05.197680    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:51:05.627941 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.627859    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"]
Oct 30 19:51:05.633836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.633802    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:05.687607 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.687582    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-86e4a7-serving-cert\""
Oct 30 19:51:05.705489 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.705459    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76483a35-32ab-46c0-9223-4ecfe57bff49-openshift-service-ca\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:05.705609 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.705594    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cj5c\" (UniqueName: \"kubernetes.io/projected/76483a35-32ab-46c0-9223-4ecfe57bff49-kube-api-access-8cj5c\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:05.705754 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.705733    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-metrics-certs\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:05.705860 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.705808    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-maas-api-tls\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:05.739979 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.739948    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"]
Oct 30 19:51:05.807415 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.807380    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-metrics-certs\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:05.807584 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.807433    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-maas-api-tls\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:05.807584 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.807504    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76483a35-32ab-46c0-9223-4ecfe57bff49-openshift-service-ca\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:05.807584 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.807583    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cj5c\" (UniqueName: \"kubernetes.io/projected/76483a35-32ab-46c0-9223-4ecfe57bff49-kube-api-access-8cj5c\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:05.807762 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:51:05.807603    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-serving-cert: secret "maas-api-e2e-api-a-86e4a7-serving-cert" not found
Oct 30 19:51:05.807762 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:51:05.807678    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-maas-api-tls podName:76483a35-32ab-46c0-9223-4ecfe57bff49 nodeName:}" failed. No retries permitted until 2026-10-30 19:51:06.307657292 +0000 UTC m=+2113.518874422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-maas-api-tls") pod "maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2" (UID: "76483a35-32ab-46c0-9223-4ecfe57bff49") : secret "maas-api-e2e-api-a-86e4a7-serving-cert" not found
Oct 30 19:51:05.808175 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.808159    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76483a35-32ab-46c0-9223-4ecfe57bff49-openshift-service-ca\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:05.809828 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.809801    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-metrics-certs\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:05.864432 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:05.864409    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cj5c\" (UniqueName: \"kubernetes.io/projected/76483a35-32ab-46c0-9223-4ecfe57bff49-kube-api-access-8cj5c\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:06.314234 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:06.314202    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-maas-api-tls\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:06.316715 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:06.316694    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-maas-api-tls\") pod \"maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:06.548493 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:06.548467    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:06.961276 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:06.961245    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"]
Oct 30 19:51:06.962626 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:51:06.962601    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76483a35_32ab_46c0_9223_4ecfe57bff49.slice/crio-39a232fa212414891f29cfa8659be961aba45cbb24d8e03174ba34d891876696 WatchSource:0}: Error finding container 39a232fa212414891f29cfa8659be961aba45cbb24d8e03174ba34d891876696: Status 404 returned error can't find the container with id 39a232fa212414891f29cfa8659be961aba45cbb24d8e03174ba34d891876696
Oct 30 19:51:06.964276 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:06.964258    4420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:51:07.392150 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:07.392118    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2" event={"ID":"76483a35-32ab-46c0-9223-4ecfe57bff49","Type":"ContainerStarted","Data":"39a232fa212414891f29cfa8659be961aba45cbb24d8e03174ba34d891876696"}
Oct 30 19:51:08.400751 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:08.400721    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2" event={"ID":"76483a35-32ab-46c0-9223-4ecfe57bff49","Type":"ContainerStarted","Data":"c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73"}
Oct 30 19:51:08.401185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:08.400842    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:08.517247 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:08.517200    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2" podStartSLOduration=2.719626671 podStartE2EDuration="3.517185316s" totalImagesPullingTime="797.558645ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:08.514884923 +0000 UTC m=+2115.726102068" watchObservedRunningTime="2026-10-30 19:51:08.517185316 +0000 UTC m=+2115.728402460"
Oct 30 19:51:14.411831 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:14.411791    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:51:20.014589 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.014549    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"]
Oct 30 19:51:20.018964 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.018946    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.076417 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.076389    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"]
Oct 30 19:51:20.093078 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.093047    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-maas-api-tls\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.093224 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.093177    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzxzx\" (UniqueName: \"kubernetes.io/projected/8d3ad8c3-3408-4e6b-b728-450d3f14466d-kube-api-access-xzxzx\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.093379 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.093364    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ad8c3-3408-4e6b-b728-450d3f14466d-openshift-service-ca\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.093445 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.093397    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-metrics-certs\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.121295 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.121269    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cb14ed-serving-cert\""
Oct 30 19:51:20.194869 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.194810    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ad8c3-3408-4e6b-b728-450d3f14466d-openshift-service-ca\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.195027 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.194885    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-metrics-certs\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.195027 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.194937    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-maas-api-tls\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.195027 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.194992    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzxzx\" (UniqueName: \"kubernetes.io/projected/8d3ad8c3-3408-4e6b-b728-450d3f14466d-kube-api-access-xzxzx\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.195201 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:51:20.195104    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-serving-cert: secret "maas-api-e2e-api-a-cb14ed-serving-cert" not found
Oct 30 19:51:20.195201 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:51:20.195189    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-maas-api-tls podName:8d3ad8c3-3408-4e6b-b728-450d3f14466d nodeName:}" failed. No retries permitted until 2026-10-30 19:51:20.695166883 +0000 UTC m=+2127.906384019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-maas-api-tls") pod "maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv" (UID: "8d3ad8c3-3408-4e6b-b728-450d3f14466d") : secret "maas-api-e2e-api-a-cb14ed-serving-cert" not found
Oct 30 19:51:20.195402 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.195384    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ad8c3-3408-4e6b-b728-450d3f14466d-openshift-service-ca\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.197367 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.197346    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-metrics-certs\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.251756 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.251734    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzxzx\" (UniqueName: \"kubernetes.io/projected/8d3ad8c3-3408-4e6b-b728-450d3f14466d-kube-api-access-xzxzx\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.702939 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.702904    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-maas-api-tls\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.705414 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.705392    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-maas-api-tls\") pod \"maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:20.933080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:20.933052    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:21.312031 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:21.312000    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"]
Oct 30 19:51:21.313564 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:51:21.313541    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3ad8c3_3408_4e6b_b728_450d3f14466d.slice/crio-a7f276a6a5f0c5c67c15776e23f5deb7d768cb0f4e3ae7f41df8b585bdee587d WatchSource:0}: Error finding container a7f276a6a5f0c5c67c15776e23f5deb7d768cb0f4e3ae7f41df8b585bdee587d: Status 404 returned error can't find the container with id a7f276a6a5f0c5c67c15776e23f5deb7d768cb0f4e3ae7f41df8b585bdee587d
Oct 30 19:51:21.483196 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:21.483159    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv" event={"ID":"8d3ad8c3-3408-4e6b-b728-450d3f14466d","Type":"ContainerStarted","Data":"a7f276a6a5f0c5c67c15776e23f5deb7d768cb0f4e3ae7f41df8b585bdee587d"}
Oct 30 19:51:22.491185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:22.491145    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv" event={"ID":"8d3ad8c3-3408-4e6b-b728-450d3f14466d","Type":"ContainerStarted","Data":"21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee"}
Oct 30 19:51:22.491569 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:22.491208    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:22.606674 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:22.606616    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv" podStartSLOduration=2.793816043 podStartE2EDuration="3.606600923s" totalImagesPullingTime="812.78488ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:22.604218385 +0000 UTC m=+2129.815435529" watchObservedRunningTime="2026-10-30 19:51:22.606600923 +0000 UTC m=+2129.817818114"
Oct 30 19:51:28.501622 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:28.501589    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:51:35.197525 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:51:35.197489    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:51:38.898221 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:38.898193    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"]
Oct 30 19:51:38.902831 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:38.902791    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:38.945081 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:38.945054    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-maas-api-tls\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:38.945222 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:38.945179    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-metrics-certs\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:38.945222 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:38.945216    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7pd\" (UniqueName: \"kubernetes.io/projected/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-kube-api-access-rk7pd\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:38.945319 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:38.945257    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-openshift-service-ca\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:39.004175 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.004138    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-daa1ac-serving-cert\""
Oct 30 19:51:39.007878 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.007850    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"]
Oct 30 19:51:39.046841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.046783    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-maas-api-tls\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:39.046999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.046928    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-metrics-certs\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:39.046999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.046963    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7pd\" (UniqueName: \"kubernetes.io/projected/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-kube-api-access-rk7pd\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:39.046999 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:51:39.046990    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-serving-cert: secret "maas-api-e2e-inf-a-daa1ac-serving-cert" not found
Oct 30 19:51:39.047172 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:51:39.047061    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-maas-api-tls podName:6fa6ea6a-ca2f-49e7-9a94-75a001e56a42 nodeName:}" failed. No retries permitted until 2026-10-30 19:51:39.547039594 +0000 UTC m=+2146.758256728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-maas-api-tls") pod "maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh" (UID: "6fa6ea6a-ca2f-49e7-9a94-75a001e56a42") : secret "maas-api-e2e-inf-a-daa1ac-serving-cert" not found
Oct 30 19:51:39.047172 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.046996    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-openshift-service-ca\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:39.047994 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.047967    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-openshift-service-ca\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:39.050135 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.050109    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-metrics-certs\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:39.116287 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.116259    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7pd\" (UniqueName: \"kubernetes.io/projected/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-kube-api-access-rk7pd\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:39.556308 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.556275    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-maas-api-tls\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:39.558919 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.558895    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-maas-api-tls\") pod \"maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:39.817248 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:39.817180    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:40.193678 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:40.193651    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"]
Oct 30 19:51:40.194768 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:51:40.194744    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa6ea6a_ca2f_49e7_9a94_75a001e56a42.slice/crio-db2d0b60ed30b11866497fb2d49ffd85eae65ed3e7591b29e5a0994807166e67 WatchSource:0}: Error finding container db2d0b60ed30b11866497fb2d49ffd85eae65ed3e7591b29e5a0994807166e67: Status 404 returned error can't find the container with id db2d0b60ed30b11866497fb2d49ffd85eae65ed3e7591b29e5a0994807166e67
Oct 30 19:51:40.607511 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:40.607417    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh" event={"ID":"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42","Type":"ContainerStarted","Data":"db2d0b60ed30b11866497fb2d49ffd85eae65ed3e7591b29e5a0994807166e67"}
Oct 30 19:51:41.615506 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:41.615463    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh" event={"ID":"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42","Type":"ContainerStarted","Data":"08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7"}
Oct 30 19:51:41.615925 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:41.615568    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:41.765408 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:41.765356    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh" podStartSLOduration=2.894652711 podStartE2EDuration="3.765342093s" totalImagesPullingTime="870.689382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:41.764576123 +0000 UTC m=+2148.975793280" watchObservedRunningTime="2026-10-30 19:51:41.765342093 +0000 UTC m=+2148.976559238"
Oct 30 19:51:47.629651 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:47.629577    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:51:51.240048 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:51.240017    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"]
Oct 30 19:51:51.240568 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:51.240230    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerName="istio-proxy" containerID="cri-o://033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1" gracePeriod=30
Oct 30 19:51:51.283428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:51.283400    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"]
Oct 30 19:51:51.283680 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:51.283658    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" podUID="9c6ccb45-a25c-482a-9698-bada65a818bc" containerName="istio-proxy" containerID="cri-o://42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2" gracePeriod=30
Oct 30 19:51:55.934980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:55.934939    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"]
Oct 30 19:51:55.935459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:55.935178    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy" containerID="cri-o://408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba" gracePeriod=30
Oct 30 19:51:56.582706 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.582681    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:51:56.620170 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.620147    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:51:56.694054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694024    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-podinfo\") pod \"f8d23027-83a7-4090-a3f6-2d22c3671881\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") "
Oct 30 19:51:56.694197 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694072    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-kube-api-access-fz92r\" (UniqueName: \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-kube-api-access-fz92r\") pod \"f8d23027-83a7-4090-a3f6-2d22c3671881\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") "
Oct 30 19:51:56.694197 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694114    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-envoy\") pod \"f8d23027-83a7-4090-a3f6-2d22c3671881\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") "
Oct 30 19:51:56.694197 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694187    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-certs\") pod \"f8d23027-83a7-4090-a3f6-2d22c3671881\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") "
Oct 30 19:51:56.694350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694238    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d23027-83a7-4090-a3f6-2d22c3671881-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d23027-83a7-4090-a3f6-2d22c3671881-istiod-ca-cert\") pod \"f8d23027-83a7-4090-a3f6-2d22c3671881\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") "
Oct 30 19:51:56.694350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694267    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-socket\") pod \"f8d23027-83a7-4090-a3f6-2d22c3671881\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") "
Oct 30 19:51:56.694350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694325    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-credential-socket\") pod \"f8d23027-83a7-4090-a3f6-2d22c3671881\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") "
Oct 30 19:51:56.694511 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694387    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-token\" (UniqueName: \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-token\") pod \"f8d23027-83a7-4090-a3f6-2d22c3671881\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") "
Oct 30 19:51:56.694511 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694442    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-certs" pod "f8d23027-83a7-4090-a3f6-2d22c3671881" (UID: "f8d23027-83a7-4090-a3f6-2d22c3671881"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:56.694626 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694543    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-data\") pod \"f8d23027-83a7-4090-a3f6-2d22c3671881\" (UID: \"f8d23027-83a7-4090-a3f6-2d22c3671881\") "
Oct 30 19:51:56.694626 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694584    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-socket" pod "f8d23027-83a7-4090-a3f6-2d22c3671881" (UID: "f8d23027-83a7-4090-a3f6-2d22c3671881"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:56.694737 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694664    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-credential-socket" pod "f8d23027-83a7-4090-a3f6-2d22c3671881" (UID: "f8d23027-83a7-4090-a3f6-2d22c3671881"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:56.694973 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.694905    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-data" pod "f8d23027-83a7-4090-a3f6-2d22c3671881" (UID: "f8d23027-83a7-4090-a3f6-2d22c3671881"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:56.695363 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.695337    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d23027-83a7-4090-a3f6-2d22c3671881-istiod-ca-cert" pod "f8d23027-83a7-4090-a3f6-2d22c3671881" (UID: "f8d23027-83a7-4090-a3f6-2d22c3671881"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:56.695513 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.695490    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.695594 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.695521    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.695594 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.695538    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8d23027-83a7-4090-a3f6-2d22c3671881-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.695594 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.695553    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.695594 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.695569    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.696644 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.696625    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-envoy" pod "f8d23027-83a7-4090-a3f6-2d22c3671881" (UID: "f8d23027-83a7-4090-a3f6-2d22c3671881"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:56.696752 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.696712    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-token" pod "f8d23027-83a7-4090-a3f6-2d22c3671881" (UID: "f8d23027-83a7-4090-a3f6-2d22c3671881"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:56.696904 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.696885    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-podinfo" pod "f8d23027-83a7-4090-a3f6-2d22c3671881" (UID: "f8d23027-83a7-4090-a3f6-2d22c3671881"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:51:56.696991 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.696927    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-kube-api-access-fz92r" pod "f8d23027-83a7-4090-a3f6-2d22c3671881" (UID: "f8d23027-83a7-4090-a3f6-2d22c3671881"). InnerVolumeSpecName "kube-api-access-fz92r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:56.720195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.720169    4420 generic.go:356] "Generic (PLEG): container finished" podID="9c6ccb45-a25c-482a-9698-bada65a818bc" containerID="42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2" exitCode=0
Oct 30 19:51:56.720297 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.720214    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"
Oct 30 19:51:56.720297 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.720248    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" event={"ID":"9c6ccb45-a25c-482a-9698-bada65a818bc","Type":"ContainerDied","Data":"42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2"}
Oct 30 19:51:56.720394 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.720328    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t" event={"ID":"9c6ccb45-a25c-482a-9698-bada65a818bc","Type":"ContainerDied","Data":"5942cebb051ba9f7ef89e1674cf95813059e471a6d8e5391efb2fdaab835be3f"}
Oct 30 19:51:56.720394 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.720348    4420 scope.go:122] "RemoveContainer" containerID="42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2"
Oct 30 19:51:56.721710 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.721694    4420 generic.go:356] "Generic (PLEG): container finished" podID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerID="033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1" exitCode=0
Oct 30 19:51:56.721791 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.721779    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" event={"ID":"f8d23027-83a7-4090-a3f6-2d22c3671881","Type":"ContainerDied","Data":"033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1"}
Oct 30 19:51:56.721880 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.721763    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"
Oct 30 19:51:56.721966 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.721826    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq" event={"ID":"f8d23027-83a7-4090-a3f6-2d22c3671881","Type":"ContainerDied","Data":"34e657256cb7cd115690171c490f3ab31c31d39c1f32e4939baedbe830ca985e"}
Oct 30 19:51:56.735027 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.735009    4420 scope.go:122] "RemoveContainer" containerID="42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2"
Oct 30 19:51:56.735307 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:51:56.735289    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2\": container with ID starting with 42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2 not found: ID does not exist" containerID="42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2"
Oct 30 19:51:56.735356 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.735317    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2"} err="failed to get container status \"42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2\": rpc error: code = NotFound desc = could not find container \"42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2\": container with ID starting with 42640cd5d4a2a104483dc921f3b706a127e221ed1f072163ddf8e1003f120dd2 not found: ID does not exist"
Oct 30 19:51:56.735356 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.735333    4420 scope.go:122] "RemoveContainer" containerID="033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1"
Oct 30 19:51:56.744342 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.744327    4420 scope.go:122] "RemoveContainer" containerID="033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1"
Oct 30 19:51:56.744610 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:51:56.744593    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1\": container with ID starting with 033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1 not found: ID does not exist" containerID="033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1"
Oct 30 19:51:56.744659 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.744619    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1"} err="failed to get container status \"033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1\": rpc error: code = NotFound desc = could not find container \"033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1\": container with ID starting with 033cbf97dcf1fbf5a3ce316e31327b1a1d391cc1cd244417515e002f3ae934f1 not found: ID does not exist"
Oct 30 19:51:56.796928 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.796849    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6ccb45-a25c-482a-9698-bada65a818bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ccb45-a25c-482a-9698-bada65a818bc-istiod-ca-cert\") pod \"9c6ccb45-a25c-482a-9698-bada65a818bc\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") "
Oct 30 19:51:56.796928 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.796905    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-data\") pod \"9c6ccb45-a25c-482a-9698-bada65a818bc\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") "
Oct 30 19:51:56.797136 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.796933    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-kube-api-access-fxhsf\" (UniqueName: \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-kube-api-access-fxhsf\") pod \"9c6ccb45-a25c-482a-9698-bada65a818bc\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") "
Oct 30 19:51:56.797136 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.796955    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-socket\") pod \"9c6ccb45-a25c-482a-9698-bada65a818bc\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") "
Oct 30 19:51:56.797136 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797062    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-envoy\") pod \"9c6ccb45-a25c-482a-9698-bada65a818bc\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") "
Oct 30 19:51:56.797136 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797092    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-podinfo\") pod \"9c6ccb45-a25c-482a-9698-bada65a818bc\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") "
Oct 30 19:51:56.797350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797192    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-data" pod "9c6ccb45-a25c-482a-9698-bada65a818bc" (UID: "9c6ccb45-a25c-482a-9698-bada65a818bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:56.797350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797225    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-token\" (UniqueName: \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-token\") pod \"9c6ccb45-a25c-482a-9698-bada65a818bc\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") "
Oct 30 19:51:56.797350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797265    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-credential-socket\") pod \"9c6ccb45-a25c-482a-9698-bada65a818bc\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") "
Oct 30 19:51:56.797350 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797309    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-certs\") pod \"9c6ccb45-a25c-482a-9698-bada65a818bc\" (UID: \"9c6ccb45-a25c-482a-9698-bada65a818bc\") "
Oct 30 19:51:56.797593 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797266    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-socket" pod "9c6ccb45-a25c-482a-9698-bada65a818bc" (UID: "9c6ccb45-a25c-482a-9698-bada65a818bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:56.797593 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797355    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6ccb45-a25c-482a-9698-bada65a818bc-istiod-ca-cert" pod "9c6ccb45-a25c-482a-9698-bada65a818bc" (UID: "9c6ccb45-a25c-482a-9698-bada65a818bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:51:56.797593 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797476    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-credential-socket" pod "9c6ccb45-a25c-482a-9698-bada65a818bc" (UID: "9c6ccb45-a25c-482a-9698-bada65a818bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:56.797757 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797678    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-certs" pod "9c6ccb45-a25c-482a-9698-bada65a818bc" (UID: "9c6ccb45-a25c-482a-9698-bada65a818bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:56.797992 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797971    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.798080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.797998    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.798080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.798012    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.798080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.798025    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6ccb45-a25c-482a-9698-bada65a818bc-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.798080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.798038    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.798080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.798051    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.798080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.798064    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.798080 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.798077    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz92r\" (UniqueName: \"kubernetes.io/projected/f8d23027-83a7-4090-a3f6-2d22c3671881-kube-api-access-fz92r\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.798370 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.798090    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8d23027-83a7-4090-a3f6-2d22c3671881-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.799444 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.799424    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-envoy" pod "9c6ccb45-a25c-482a-9698-bada65a818bc" (UID: "9c6ccb45-a25c-482a-9698-bada65a818bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:51:56.799528 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.799494    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-token" pod "9c6ccb45-a25c-482a-9698-bada65a818bc" (UID: "9c6ccb45-a25c-482a-9698-bada65a818bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:56.799828 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.799796    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-kube-api-access-fxhsf" pod "9c6ccb45-a25c-482a-9698-bada65a818bc" (UID: "9c6ccb45-a25c-482a-9698-bada65a818bc"). InnerVolumeSpecName "kube-api-access-fxhsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:51:56.799884 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.799806    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-podinfo" pod "9c6ccb45-a25c-482a-9698-bada65a818bc" (UID: "9c6ccb45-a25c-482a-9698-bada65a818bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:51:56.899381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.899348    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.899381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.899378    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxhsf\" (UniqueName: \"kubernetes.io/projected/9c6ccb45-a25c-482a-9698-bada65a818bc-kube-api-access-fxhsf\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.899381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.899388    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.899686 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.899397    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6ccb45-a25c-482a-9698-bada65a818bc-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:51:56.981347 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.981318    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"]
Oct 30 19:51:56.984331 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:56.984307    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-b84jq"]
Oct 30 19:51:57.135235 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:57.135168    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"]
Oct 30 19:51:57.141005 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:57.140979    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cf9b6f-openshift-default-56db6769f7-gb58t"]
Oct 30 19:51:58.896628 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:58.896598    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6ccb45-a25c-482a-9698-bada65a818bc" path="/var/lib/kubelet/pods/9c6ccb45-a25c-482a-9698-bada65a818bc/volumes"
Oct 30 19:51:58.897092 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:51:58.897078    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" path="/var/lib/kubelet/pods/f8d23027-83a7-4090-a3f6-2d22c3671881/volumes"
Oct 30 19:52:00.061502 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:00.061466    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:52:00.061866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:00.061525    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:52:01.212887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.212834    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:52:01.258336 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258313    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-data\") pod \"d4dce860-24df-401c-a725-60bd751e2203\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") "
Oct 30 19:52:01.258463 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258423    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-istio-token\" (UniqueName: \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-istio-token\") pod \"d4dce860-24df-401c-a725-60bd751e2203\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") "
Oct 30 19:52:01.258511 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258464    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-certs\") pod \"d4dce860-24df-401c-a725-60bd751e2203\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") "
Oct 30 19:52:01.258561 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258539    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-credential-socket\") pod \"d4dce860-24df-401c-a725-60bd751e2203\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") "
Oct 30 19:52:01.258611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258574    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4dce860-24df-401c-a725-60bd751e2203-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4dce860-24df-401c-a725-60bd751e2203-istio-podinfo\") pod \"d4dce860-24df-401c-a725-60bd751e2203\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") "
Oct 30 19:52:01.258687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258566    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-data" pod "d4dce860-24df-401c-a725-60bd751e2203" (UID: "d4dce860-24df-401c-a725-60bd751e2203"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:01.258747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258727    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-envoy\") pod \"d4dce860-24df-401c-a725-60bd751e2203\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") "
Oct 30 19:52:01.258804 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258773    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-kube-api-access-s7rcf\" (UniqueName: \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-kube-api-access-s7rcf\") pod \"d4dce860-24df-401c-a725-60bd751e2203\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") "
Oct 30 19:52:01.258804 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258790    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-certs" pod "d4dce860-24df-401c-a725-60bd751e2203" (UID: "d4dce860-24df-401c-a725-60bd751e2203"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:01.258932 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258841    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4dce860-24df-401c-a725-60bd751e2203-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4dce860-24df-401c-a725-60bd751e2203-istiod-ca-cert\") pod \"d4dce860-24df-401c-a725-60bd751e2203\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") "
Oct 30 19:52:01.258932 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258891    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-socket\") pod \"d4dce860-24df-401c-a725-60bd751e2203\" (UID: \"d4dce860-24df-401c-a725-60bd751e2203\") "
Oct 30 19:52:01.258932 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.258915    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-credential-socket" pod "d4dce860-24df-401c-a725-60bd751e2203" (UID: "d4dce860-24df-401c-a725-60bd751e2203"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:01.259359 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.259310    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4dce860-24df-401c-a725-60bd751e2203-istiod-ca-cert" pod "d4dce860-24df-401c-a725-60bd751e2203" (UID: "d4dce860-24df-401c-a725-60bd751e2203"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:52:01.259359 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.259333    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-socket" pod "d4dce860-24df-401c-a725-60bd751e2203" (UID: "d4dce860-24df-401c-a725-60bd751e2203"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:01.259520 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.259497    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:01.259520 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.259515    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4dce860-24df-401c-a725-60bd751e2203-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:01.259665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.259529    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:01.259665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.259543    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:01.259665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.259556    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:01.261338 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.261315    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-envoy" pod "d4dce860-24df-401c-a725-60bd751e2203" (UID: "d4dce860-24df-401c-a725-60bd751e2203"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:52:01.261431 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.261342    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-istio-token" pod "d4dce860-24df-401c-a725-60bd751e2203" (UID: "d4dce860-24df-401c-a725-60bd751e2203"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:01.261491 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.261430    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-kube-api-access-s7rcf" pod "d4dce860-24df-401c-a725-60bd751e2203" (UID: "d4dce860-24df-401c-a725-60bd751e2203"). InnerVolumeSpecName "kube-api-access-s7rcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:52:01.261549 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.261529    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4dce860-24df-401c-a725-60bd751e2203-istio-podinfo" pod "d4dce860-24df-401c-a725-60bd751e2203" (UID: "d4dce860-24df-401c-a725-60bd751e2203"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:52:01.361339 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.361313    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:01.361339 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.361336    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4dce860-24df-401c-a725-60bd751e2203-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:01.361488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.361345    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4dce860-24df-401c-a725-60bd751e2203-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:01.361488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.361357    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7rcf\" (UniqueName: \"kubernetes.io/projected/d4dce860-24df-401c-a725-60bd751e2203-kube-api-access-s7rcf\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:52:01.756282 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.756249    4420 generic.go:356] "Generic (PLEG): container finished" podID="d4dce860-24df-401c-a725-60bd751e2203" containerID="408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba" exitCode=0
Oct 30 19:52:01.756484 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.756295    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"
Oct 30 19:52:01.756484 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.756336    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" event={"ID":"d4dce860-24df-401c-a725-60bd751e2203","Type":"ContainerDied","Data":"408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba"}
Oct 30 19:52:01.756484 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.756399    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh" event={"ID":"d4dce860-24df-401c-a725-60bd751e2203","Type":"ContainerDied","Data":"65cadd8354191cfb2ec3fcc894f7e91b5171b8a405e43027bda58b32b4e30e4a"}
Oct 30 19:52:01.756484 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.756428    4420 scope.go:122] "RemoveContainer" containerID="408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba"
Oct 30 19:52:01.767490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.767472    4420 scope.go:122] "RemoveContainer" containerID="408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba"
Oct 30 19:52:01.767747 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:52:01.767729    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba\": container with ID starting with 408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba not found: ID does not exist" containerID="408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba"
Oct 30 19:52:01.767796 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.767756    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba"} err="failed to get container status \"408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba\": rpc error: code = NotFound desc = could not find container \"408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba\": container with ID starting with 408954c7e128e96aa6eb59fe80201d6b2540ae7aa73fca6db8de71eb9312d7ba not found: ID does not exist"
Oct 30 19:52:01.920957 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.920926    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"]
Oct 30 19:52:01.971117 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:01.971093    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-4k4gh"]
Oct 30 19:52:02.898329 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:02.898290    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4dce860-24df-401c-a725-60bd751e2203" path="/var/lib/kubelet/pods/d4dce860-24df-401c-a725-60bd751e2203/volumes"
Oct 30 19:52:05.197238 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:52:05.197204    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:52:07.612214 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.612183    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"]
Oct 30 19:52:07.612793 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.612780    4420 cpu_manager.go:414] "containerMap: removing container" podUID="9c6ccb45-a25c-482a-9698-bada65a818bc" containerName="istio-proxy"
Oct 30 19:52:07.612857 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.612795    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6ccb45-a25c-482a-9698-bada65a818bc" containerName="istio-proxy"
Oct 30 19:52:07.612857 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.612844    4420 cpu_manager.go:414] "containerMap: removing container" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy"
Oct 30 19:52:07.612857 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.612850    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy"
Oct 30 19:52:07.612953 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.612867    4420 cpu_manager.go:414] "containerMap: removing container" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerName="istio-proxy"
Oct 30 19:52:07.612953 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.612872    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerName="istio-proxy"
Oct 30 19:52:07.613012 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.612971    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4dce860-24df-401c-a725-60bd751e2203" containerName="istio-proxy"
Oct 30 19:52:07.613012 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.612981    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d23027-83a7-4090-a3f6-2d22c3671881" containerName="istio-proxy"
Oct 30 19:52:07.613012 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.612990    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6ccb45-a25c-482a-9698-bada65a818bc" containerName="istio-proxy"
Oct 30 19:52:07.617550 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.617534    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:07.669241 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.669218    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6873e0-serving-cert\""
Oct 30 19:52:07.722444 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.722414    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"]
Oct 30 19:52:07.737290 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.737263    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4kjj\" (UniqueName: \"kubernetes.io/projected/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-kube-api-access-x4kjj\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:07.737422 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.737398    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-maas-api-tls\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:07.737461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.737424    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-metrics-certs\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:07.737639 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.737616    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:07.838717 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.838677    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:07.838939 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.838805    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4kjj\" (UniqueName: \"kubernetes.io/projected/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-kube-api-access-x4kjj\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:07.839033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.838981    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-maas-api-tls\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:07.839033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.839011    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-metrics-certs\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:07.839168 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:52:07.839140    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-serving-cert: secret "maas-api-e2e-inf-a-6873e0-serving-cert" not found
Oct 30 19:52:07.839247 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:52:07.839235    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-maas-api-tls podName:6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c nodeName:}" failed. No retries permitted until 2026-10-30 19:52:08.339215013 +0000 UTC m=+2175.550432151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-maas-api-tls") pod "maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm" (UID: "6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c") : secret "maas-api-e2e-inf-a-6873e0-serving-cert" not found
Oct 30 19:52:07.839427 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.839409    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:07.841516 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.841490    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-metrics-certs\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:07.898671 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:07.898643    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4kjj\" (UniqueName: \"kubernetes.io/projected/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-kube-api-access-x4kjj\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:08.346050 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:08.345973    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-maas-api-tls\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:08.348523 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:08.348496    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-maas-api-tls\") pod \"maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:08.533297 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:08.533272    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:08.914403 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:08.914375    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"]
Oct 30 19:52:08.915147 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:52:08.915122    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd1e8d2_cff8_4a83_a24d_7e5e19d8ad9c.slice/crio-aeb1a6b14e60b61adbc433b5a6584e317e21d280ecae0ee2cefa4e4587b4acdc WatchSource:0}: Error finding container aeb1a6b14e60b61adbc433b5a6584e317e21d280ecae0ee2cefa4e4587b4acdc: Status 404 returned error can't find the container with id aeb1a6b14e60b61adbc433b5a6584e317e21d280ecae0ee2cefa4e4587b4acdc
Oct 30 19:52:09.817896 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:09.817852    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm" event={"ID":"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c","Type":"ContainerStarted","Data":"896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15"}
Oct 30 19:52:09.818042 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:09.817914    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm" event={"ID":"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c","Type":"ContainerStarted","Data":"aeb1a6b14e60b61adbc433b5a6584e317e21d280ecae0ee2cefa4e4587b4acdc"}
Oct 30 19:52:09.818042 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:09.817937    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:09.936747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:09.936677    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm" podStartSLOduration=2.147444652 podStartE2EDuration="2.936660567s" totalImagesPullingTime="789.215915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:09.936236762 +0000 UTC m=+2177.147453908" watchObservedRunningTime="2026-10-30 19:52:09.936660567 +0000 UTC m=+2177.147877715"
Oct 30 19:52:15.829929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:15.829903    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:52:20.117012 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:20.116973    4420 patch_prober.go:29] interesting pod/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:52:20.117482 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:20.117063    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:52:22.967340 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:22.967300    4420 generic.go:356] "Generic (PLEG): container finished" podID="415f5443-053d-4e88-814d-91952beb32f5" containerID="af82ba18af013c0ce5d8a6004368c71f9556a32f48e2ec64559596ae2b03d4f6" exitCode=0
Oct 30 19:52:22.967340 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:22.967340    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" event={"ID":"415f5443-053d-4e88-814d-91952beb32f5","Type":"ContainerDied","Data":"af82ba18af013c0ce5d8a6004368c71f9556a32f48e2ec64559596ae2b03d4f6"}
Oct 30 19:52:22.967960 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:22.967938    4420 scope.go:122] "RemoveContainer" containerID="af82ba18af013c0ce5d8a6004368c71f9556a32f48e2ec64559596ae2b03d4f6"
Oct 30 19:52:22.970195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:22.970165    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:22.970301 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:22.970237    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:22.970301 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:22.970283    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:23.995560 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:23.995520    4420 generic.go:356] "Generic (PLEG): container finished" podID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerID="502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19" exitCode=0
Oct 30 19:52:23.996071 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:23.995566    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" event={"ID":"b74aa3ef-91cc-4ece-b74e-c3332572e110","Type":"ContainerDied","Data":"502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19"}
Oct 30 19:52:23.996165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:23.996146    4420 scope.go:122] "RemoveContainer" containerID="502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19"
Oct 30 19:52:23.998115 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:23.998081    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" event={"ID":"415f5443-053d-4e88-814d-91952beb32f5","Type":"ContainerStarted","Data":"63764c9cf3a7abe8d7234808caf1716a79b160ae58509226a05b06509ed00381"}
Oct 30 19:52:23.998249 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:23.998091    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:23.998249 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:23.998234    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:23.998341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:23.998283    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:24.105397 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:24.105375    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:52:24.105862 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:24.105628    4420 patch_prober.go:29] interesting pod/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:24.105862 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:24.105676    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:52:25.014520 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:25.013304    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" event={"ID":"b74aa3ef-91cc-4ece-b74e-c3332572e110","Type":"ContainerStarted","Data":"b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead"}
Oct 30 19:52:25.106269 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:25.106233    4420 patch_prober.go:29] interesting pod/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:25.106561 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:25.106532    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:52:25.975195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:25.975156    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:52:25.975628 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:25.975605    4420 patch_prober.go:29] interesting pod/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:25.975787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:25.975763    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:52:26.111841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:26.111207    4420 patch_prober.go:29] interesting pod/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:26.111841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:26.111295    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:52:26.919344 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:26.919313    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:52:26.974842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:26.974258    4420 patch_prober.go:29] interesting pod/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:26.974842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:26.974313    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:52:27.111844 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:27.105256    4420 patch_prober.go:29] interesting pod/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:27.111844 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:27.105331    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:52:27.984841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:27.976199    4420 patch_prober.go:29] interesting pod/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:27.984841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:27.976260    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:52:28.109842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:28.105355    4420 patch_prober.go:29] interesting pod/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:28.109842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:28.105431    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:52:28.979842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:28.979189    4420 patch_prober.go:29] interesting pod/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:28.979842 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:28.979265    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:52:29.104648 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:29.104610    4420 patch_prober.go:29] interesting pod/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Oct 30 19:52:29.104856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:29.104680    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Oct 30 19:52:29.976050 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:29.976008    4420 patch_prober.go:29] interesting pod/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:52:29.976241 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:29.976081    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 19:52:30.110055 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:30.110024    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:52:30.110521 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:30.110502    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:52:30.111184 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:30.111165    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:52:30.978623 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:30.978595    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:52:30.979548 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:30.979526    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:52:35.196798 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:52:35.196719    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:52:42.779929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.779896    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"]
Oct 30 19:52:42.790362 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.790336    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.869849 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.868646    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-istio-token\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.889997 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.888939    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-credential-socket\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.889997 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.889055    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-envoy\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.889997 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.889183    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjsn\" (UniqueName: \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-kube-api-access-7bjsn\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.889997 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.889250    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-socket\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.889997 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.889279    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773cf534-eeb3-43e2-8616-264970478e1d-istio-podinfo\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.889997 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.889344    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-certs\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.889997 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.889438    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-data\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.889997 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.889472    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773cf534-eeb3-43e2-8616-264970478e1d-istiod-ca-cert\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.904548 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.904518    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"]
Oct 30 19:52:42.923944 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.923915    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1c5537-openshift-default-dockercfg-qm79l\""
Oct 30 19:52:42.990943 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.990906    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-data\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.990943 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.990949    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773cf534-eeb3-43e2-8616-264970478e1d-istiod-ca-cert\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.991201 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.990986    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-istio-token\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.991201 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.991059    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-credential-socket\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.991201 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.991112    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-envoy\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.991201 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.991191    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjsn\" (UniqueName: \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-kube-api-access-7bjsn\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.991418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.991233    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-socket\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.991418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.991273    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773cf534-eeb3-43e2-8616-264970478e1d-istio-podinfo\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.991418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.991333    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-certs\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.991418 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.991358    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-data\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.991733 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.991709    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-certs\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.992575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.992552    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773cf534-eeb3-43e2-8616-264970478e1d-istiod-ca-cert\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.992716 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.992694    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-socket\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.993009 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.992780    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-credential-socket\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.996077 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.996052    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-envoy\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:42.996204 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:42.996182    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773cf534-eeb3-43e2-8616-264970478e1d-istio-podinfo\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:43.049526 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:43.049459    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-istio-token\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:43.049844 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:43.049786    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjsn\" (UniqueName: \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-kube-api-access-7bjsn\") pod \"e2e-api-b-1c5537-openshift-default-5df844678c-z827l\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") " pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:43.105951 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:43.105917    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:43.586539 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:43.586508    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"]
Oct 30 19:52:43.589849 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:52:43.589801    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773cf534_eeb3_43e2_8616_264970478e1d.slice/crio-a3a78acac45ba946306e50f36d22687e4a4463433f50b2e6a894d5cf1d106885 WatchSource:0}: Error finding container a3a78acac45ba946306e50f36d22687e4a4463433f50b2e6a894d5cf1d106885: Status 404 returned error can't find the container with id a3a78acac45ba946306e50f36d22687e4a4463433f50b2e6a894d5cf1d106885
Oct 30 19:52:43.592587 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:43.592554    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:43.592677 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:43.592622    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:43.592677 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:43.592650    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:44.154887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:44.154008    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" event={"ID":"773cf534-eeb3-43e2-8616-264970478e1d","Type":"ContainerStarted","Data":"80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f"}
Oct 30 19:52:44.154887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:44.154070    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" event={"ID":"773cf534-eeb3-43e2-8616-264970478e1d","Type":"ContainerStarted","Data":"a3a78acac45ba946306e50f36d22687e4a4463433f50b2e6a894d5cf1d106885"}
Oct 30 19:52:44.277260 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:44.277194    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" podStartSLOduration=2.27717304 podStartE2EDuration="2.27717304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:44.271951747 +0000 UTC m=+2211.483168892" watchObservedRunningTime="2026-10-30 19:52:44.27717304 +0000 UTC m=+2211.488390185"
Oct 30 19:52:45.106392 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:45.106355    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:45.112569 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:45.112541    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:45.163274 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:45.163240    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:45.164279 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:45.164242    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:52:47.178440 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:47.178406    4420 generic.go:356] "Generic (PLEG): container finished" podID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerID="a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1" exitCode=0
Oct 30 19:52:47.178914 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:47.178478    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" event={"ID":"ac27ddeb-8cb2-473a-b036-3f0201411622","Type":"ContainerDied","Data":"a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1"}
Oct 30 19:52:47.178988 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:47.178973    4420 scope.go:122] "RemoveContainer" containerID="a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1"
Oct 30 19:52:47.180566 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:47.180539    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:47.180639 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:47.180610    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:47.180676 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:47.180641    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:48.193460 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:48.193419    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" event={"ID":"ac27ddeb-8cb2-473a-b036-3f0201411622","Type":"ContainerStarted","Data":"81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7"}
Oct 30 19:52:48.195103 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:48.195080    4420 generic.go:356] "Generic (PLEG): container finished" podID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerID="714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553" exitCode=0
Oct 30 19:52:48.195215 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:48.195153    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" event={"ID":"72e17a94-a1f3-4273-a8f5-458b8d478a16","Type":"ContainerDied","Data":"714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553"}
Oct 30 19:52:48.195500 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:48.195486    4420 scope.go:122] "RemoveContainer" containerID="714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553"
Oct 30 19:52:48.197041 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:48.197009    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:48.197149 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:48.197090    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:48.197149 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:48.197139    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:52:48.432171 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:48.432139    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:52:48.432364 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:48.432315    4420 patch_prober.go:29] interesting pod/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:48.432425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:48.432362    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:52:49.215270 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:49.215237    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" event={"ID":"72e17a94-a1f3-4273-a8f5-458b8d478a16","Type":"ContainerStarted","Data":"3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8"}
Oct 30 19:52:49.432228 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:49.432194    4420 patch_prober.go:29] interesting pod/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:49.432397 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:49.432242    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:52:49.778841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:49.778230    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:52:49.778841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:49.778364    4420 patch_prober.go:29] interesting pod/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:52:49.778841 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:49.778421    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:52:50.432468 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:50.432429    4420 patch_prober.go:29] interesting pod/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:50.432967 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:50.432487    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:52:50.775388 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:50.775290    4420 patch_prober.go:29] interesting pod/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:52:50.775680 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:50.775625    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:52:51.432624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:51.432593    4420 patch_prober.go:29] interesting pod/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:51.433023 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:51.432649    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:52:51.773894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:51.773285    4420 patch_prober.go:29] interesting pod/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:52:51.773894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:51.773371    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:52:52.432696 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:52.432661    4420 patch_prober.go:29] interesting pod/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:52.433124 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:52.432710    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:52:52.786900 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:52.786235    4420 patch_prober.go:29] interesting pod/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:52:52.786900 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:52.786308    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:52:53.432430 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:53.432401    4420 patch_prober.go:29] interesting pod/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:52:53.432551 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:53.432452    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 19:52:53.782899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:53.779183    4420 patch_prober.go:29] interesting pod/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 30 19:52:53.782899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:53.779246    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 30 19:52:54.436720 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:54.436688    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:52:54.437198 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:54.437179    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:52:54.437886 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:54.437869    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:52:54.777873 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:54.777767    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:52:54.778219 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:54.778200    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:52:54.778895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:52:54.778877    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:53:05.197803 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:53:05.197765    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:53:17.716588 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.716550    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"]
Oct 30 19:53:17.723853 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.723806    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.825316 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.825290    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"]
Oct 30 19:53:17.840856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.835384    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-da4c62-openshift-default-dockercfg-ktmk4\""
Oct 30 19:53:17.840856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.839992    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjqjp\" (UniqueName: \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-kube-api-access-zjqjp\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.840856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.840039    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-token\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.840856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.840081    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-certs\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.840856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.840196    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-socket\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.840856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.840324    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istiod-ca-cert\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.840856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.840375    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-credential-socket\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.840856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.840454    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-envoy\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.840856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.840486    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-podinfo\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.840856 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.840579    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-data\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.941975 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.941943    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjqjp\" (UniqueName: \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-kube-api-access-zjqjp\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942168 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.941986    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-token\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942168 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942017    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-certs\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942168 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942078    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-socket\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942328 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942212    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istiod-ca-cert\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942328 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942281    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-credential-socket\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942433 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942339    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-envoy\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942433 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942366    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-podinfo\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942433 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942410    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-socket\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942723 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942476    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-data\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942723 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942497    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-certs\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942723 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942601    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-credential-socket\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942854    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-data\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.942976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.942902    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istiod-ca-cert\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.944737 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.944710    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-envoy\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.945064 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.945044    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-podinfo\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.998748 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.998679    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-token\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:17.998903 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:17.998806    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjqjp\" (UniqueName: \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-kube-api-access-zjqjp\") pod \"e2e-api-b-da4c62-openshift-default-5945967dc-zls87\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") " pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:18.127043 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:18.127008    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:18.559239 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:18.559213    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"]
Oct 30 19:53:18.559359 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:53:18.559275    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3c54ef_79b7_4c7a_bb9d_cdadfce5589e.slice/crio-8a9302865a7c3cb2967b7ee310c1cc435360976ac819e807ad75b3c246623006 WatchSource:0}: Error finding container 8a9302865a7c3cb2967b7ee310c1cc435360976ac819e807ad75b3c246623006: Status 404 returned error can't find the container with id 8a9302865a7c3cb2967b7ee310c1cc435360976ac819e807ad75b3c246623006
Oct 30 19:53:18.562033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:18.562001    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:53:18.562125 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:18.562065    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:53:18.562125 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:18.562094    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:53:19.467342 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:19.467300    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" event={"ID":"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e","Type":"ContainerStarted","Data":"7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe"}
Oct 30 19:53:19.467733 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:19.467359    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" event={"ID":"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e","Type":"ContainerStarted","Data":"8a9302865a7c3cb2967b7ee310c1cc435360976ac819e807ad75b3c246623006"}
Oct 30 19:53:19.642396 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:19.642326    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" podStartSLOduration=2.6423088310000002 podStartE2EDuration="2.642308831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:19.638798362 +0000 UTC m=+2246.850015507" watchObservedRunningTime="2026-10-30 19:53:19.642308831 +0000 UTC m=+2246.853525977"
Oct 30 19:53:20.127226 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:20.127193    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:20.132720 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:20.132691    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:20.473436 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:20.473409    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:20.474269 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:20.474252    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:53:35.197696 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:53:35.197652    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:53:56.533292 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:56.533258    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"]
Oct 30 19:53:56.533965 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:53:56.533487    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy" containerID="cri-o://e4b358af27be27af50da7e71f7591dc32213bf18ddeee0c902746abba543ccbc" gracePeriod=30
Oct 30 19:54:01.760132 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.760099    4420 generic.go:356] "Generic (PLEG): container finished" podID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerID="e4b358af27be27af50da7e71f7591dc32213bf18ddeee0c902746abba543ccbc" exitCode=0
Oct 30 19:54:01.760686 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.760141    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" event={"ID":"13fe3925-27ad-4e80-be15-e6d2e4fa826e","Type":"ContainerDied","Data":"e4b358af27be27af50da7e71f7591dc32213bf18ddeee0c902746abba543ccbc"}
Oct 30 19:54:01.808999 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.808942    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:54:01.921068 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921043    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-socket\") pod \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") "
Oct 30 19:54:01.921204 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921088    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-data\") pod \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") "
Oct 30 19:54:01.921204 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921117    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istiod-ca-cert\") pod \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") "
Oct 30 19:54:01.921204 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921167    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-certs\") pod \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") "
Oct 30 19:54:01.921372 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921210    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-envoy\") pod \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") "
Oct 30 19:54:01.921372 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921257    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-credential-socket\") pod \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") "
Oct 30 19:54:01.921372 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921328    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-socket" pod "13fe3925-27ad-4e80-be15-e6d2e4fa826e" (UID: "13fe3925-27ad-4e80-be15-e6d2e4fa826e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:01.921372 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921340    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-podinfo\") pod \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") "
Oct 30 19:54:01.921372 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921343    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-data" pod "13fe3925-27ad-4e80-be15-e6d2e4fa826e" (UID: "13fe3925-27ad-4e80-be15-e6d2e4fa826e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:01.921616 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921437    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-kube-api-access-86rpx\" (UniqueName: \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-kube-api-access-86rpx\") pod \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") "
Oct 30 19:54:01.921616 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921478    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-certs" pod "13fe3925-27ad-4e80-be15-e6d2e4fa826e" (UID: "13fe3925-27ad-4e80-be15-e6d2e4fa826e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:01.921616 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921500    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-token\" (UniqueName: \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-token\") pod \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\" (UID: \"13fe3925-27ad-4e80-be15-e6d2e4fa826e\") "
Oct 30 19:54:01.921768 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921663    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-credential-socket" pod "13fe3925-27ad-4e80-be15-e6d2e4fa826e" (UID: "13fe3925-27ad-4e80-be15-e6d2e4fa826e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:01.922031 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.921998    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istiod-ca-cert" pod "13fe3925-27ad-4e80-be15-e6d2e4fa826e" (UID: "13fe3925-27ad-4e80-be15-e6d2e4fa826e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:54:01.922121 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.922038    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.922121 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.922056    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.922121 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.922070    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.922121 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.922084    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:54:01.923768 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.923701    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-podinfo" pod "13fe3925-27ad-4e80-be15-e6d2e4fa826e" (UID: "13fe3925-27ad-4e80-be15-e6d2e4fa826e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:54:01.923908 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.923886    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-kube-api-access-86rpx" pod "13fe3925-27ad-4e80-be15-e6d2e4fa826e" (UID: "13fe3925-27ad-4e80-be15-e6d2e4fa826e"). InnerVolumeSpecName "kube-api-access-86rpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:01.924191 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.924170    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-token" pod "13fe3925-27ad-4e80-be15-e6d2e4fa826e" (UID: "13fe3925-27ad-4e80-be15-e6d2e4fa826e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:54:01.924191 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:01.924177    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-envoy" pod "13fe3925-27ad-4e80-be15-e6d2e4fa826e" (UID: "13fe3925-27ad-4e80-be15-e6d2e4fa826e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:54:02.023798 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:02.023767    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:54:02.023798 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:02.023801    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:54:02.024045 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:02.023850    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:54:02.024045 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:02.023868    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13fe3925-27ad-4e80-be15-e6d2e4fa826e-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:54:02.024045 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:02.023883    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86rpx\" (UniqueName: \"kubernetes.io/projected/13fe3925-27ad-4e80-be15-e6d2e4fa826e-kube-api-access-86rpx\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:54:02.761009 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:02.760957    4420 patch_prober.go:29] interesting pod/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:54:02.761404 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:02.761030    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:54:02.768311 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:02.768286    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"
Oct 30 19:54:02.768311 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:02.768295    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf" event={"ID":"13fe3925-27ad-4e80-be15-e6d2e4fa826e","Type":"ContainerDied","Data":"d82f388cc6d575fe84418cbfa39a7602ebc40d18e0e04994b66dc11f4d008a1e"}
Oct 30 19:54:02.768495 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:02.768373    4420 scope.go:122] "RemoveContainer" containerID="e4b358af27be27af50da7e71f7591dc32213bf18ddeee0c902746abba543ccbc"
Oct 30 19:54:02.977341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:02.977313    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"]
Oct 30 19:54:03.026148 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:03.026069    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-9hhdf"]
Oct 30 19:54:04.896447 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:04.896375    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" path="/var/lib/kubelet/pods/13fe3925-27ad-4e80-be15-e6d2e4fa826e/volumes"
Oct 30 19:54:05.197345 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:54:05.197265    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:54:19.012287 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.012252    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"]
Oct 30 19:54:19.013225 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.013208    4420 cpu_manager.go:414] "containerMap: removing container" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy"
Oct 30 19:54:19.013300 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.013227    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy"
Oct 30 19:54:19.013375 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.013365    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="13fe3925-27ad-4e80-be15-e6d2e4fa826e" containerName="istio-proxy"
Oct 30 19:54:19.017368 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.017348    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.032072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.032050    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62kqd\" (UniqueName: \"kubernetes.io/projected/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-kube-api-access-62kqd\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.032182 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.032167    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-metrics-certs\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.032319 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.032306    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-maas-api-tls\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.032358 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.032333    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.118458 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.118429    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1c5537-serving-cert\""
Oct 30 19:54:19.122027 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.122002    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"]
Oct 30 19:54:19.133371 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.133344    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-maas-api-tls\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.133490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.133385    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.133490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.133478    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62kqd\" (UniqueName: \"kubernetes.io/projected/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-kube-api-access-62kqd\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.133593 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:54:19.133489    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-serving-cert: secret "maas-api-e2e-api-b-1c5537-serving-cert" not found
Oct 30 19:54:19.133593 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:54:19.133538    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-maas-api-tls podName:7b5b0a45-21fd-4669-b6b1-d7dcb94edb09 nodeName:}" failed. No retries permitted until 2026-10-30 19:54:19.633524066 +0000 UTC m=+2306.844741216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-maas-api-tls") pod "maas-api-e2e-api-b-1c5537-b44c897d7-glj5g" (UID: "7b5b0a45-21fd-4669-b6b1-d7dcb94edb09") : secret "maas-api-e2e-api-b-1c5537-serving-cert" not found
Oct 30 19:54:19.133702 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.133612    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-metrics-certs\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.134133 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.134113    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.136218 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.136199    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-metrics-certs\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.191156 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.191124    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62kqd\" (UniqueName: \"kubernetes.io/projected/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-kube-api-access-62kqd\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.640770 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.640739    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-maas-api-tls\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.643228 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.643203    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-maas-api-tls\") pod \"maas-api-e2e-api-b-1c5537-b44c897d7-glj5g\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:19.932685 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:19.932605    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:20.311004 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:20.310974    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"]
Oct 30 19:54:20.312282 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:54:20.312257    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5b0a45_21fd_4669_b6b1_d7dcb94edb09.slice/crio-8451bf0fa23150bbbc4129d78e0020726e28e8baa1777af291c192f80a573f5c WatchSource:0}: Error finding container 8451bf0fa23150bbbc4129d78e0020726e28e8baa1777af291c192f80a573f5c: Status 404 returned error can't find the container with id 8451bf0fa23150bbbc4129d78e0020726e28e8baa1777af291c192f80a573f5c
Oct 30 19:54:20.905309 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:20.905274    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g" event={"ID":"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09","Type":"ContainerStarted","Data":"8451bf0fa23150bbbc4129d78e0020726e28e8baa1777af291c192f80a573f5c"}
Oct 30 19:54:21.914889 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:21.913846    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g" event={"ID":"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09","Type":"ContainerStarted","Data":"6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330"}
Oct 30 19:54:21.914889 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:21.914830    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:22.032274 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:22.032228    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g" podStartSLOduration=3.249969855 podStartE2EDuration="4.032215895s" totalImagesPullingTime="782.24604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:22.029757103 +0000 UTC m=+2309.240974268" watchObservedRunningTime="2026-10-30 19:54:22.032215895 +0000 UTC m=+2309.243433039"
Oct 30 19:54:28.931961 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:54:28.931934    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 19:54:35.196648 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:54:35.196610    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:55:01.641608 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.641574    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"]
Oct 30 19:55:01.647178 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.647161    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:01.718120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.718083    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:01.718311 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.718150    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db8bc9fc-79eb-466f-9f07-35f37e2798d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:01.718383 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.718304    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-metrics-certs\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:01.718500 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.718483    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhxcb\" (UniqueName: \"kubernetes.io/projected/db8bc9fc-79eb-466f-9f07-35f37e2798d4-kube-api-access-hhxcb\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:01.748150 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.748124    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b0c827-serving-cert\""
Oct 30 19:55:01.752170 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.752150    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"]
Oct 30 19:55:01.819599 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.819565    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhxcb\" (UniqueName: \"kubernetes.io/projected/db8bc9fc-79eb-466f-9f07-35f37e2798d4-kube-api-access-hhxcb\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:01.819774 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.819667    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:01.819774 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.819712    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db8bc9fc-79eb-466f-9f07-35f37e2798d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:01.819922 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.819783    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-metrics-certs\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:01.819922 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:55:01.819837    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-serving-cert: secret "maas-api-e2e-inf-b-b0c827-serving-cert" not found
Oct 30 19:55:01.819922 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:55:01.819913    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-maas-api-tls podName:db8bc9fc-79eb-466f-9f07-35f37e2798d4 nodeName:}" failed. No retries permitted until 2026-10-30 19:55:02.3198937 +0000 UTC m=+2349.531110845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-maas-api-tls") pod "maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc" (UID: "db8bc9fc-79eb-466f-9f07-35f37e2798d4") : secret "maas-api-e2e-inf-b-b0c827-serving-cert" not found
Oct 30 19:55:01.820461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.820436    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db8bc9fc-79eb-466f-9f07-35f37e2798d4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:01.822659 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.822631    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-metrics-certs\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:01.877103 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:01.877076    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhxcb\" (UniqueName: \"kubernetes.io/projected/db8bc9fc-79eb-466f-9f07-35f37e2798d4-kube-api-access-hhxcb\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:02.339002 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:02.338914    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:02.341523 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:02.341492    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-maas-api-tls\") pod \"maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:02.585739 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:02.585704    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:02.971636 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:02.971606    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"]
Oct 30 19:55:02.973350 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:55:02.973321    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb8bc9fc_79eb_466f_9f07_35f37e2798d4.slice/crio-a658fd3e2de65460c1ec464b1719d4187800a4d0e03f306bdf9d6c7973b2cfc3 WatchSource:0}: Error finding container a658fd3e2de65460c1ec464b1719d4187800a4d0e03f306bdf9d6c7973b2cfc3: Status 404 returned error can't find the container with id a658fd3e2de65460c1ec464b1719d4187800a4d0e03f306bdf9d6c7973b2cfc3
Oct 30 19:55:03.236804 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:03.236724    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc" event={"ID":"db8bc9fc-79eb-466f-9f07-35f37e2798d4","Type":"ContainerStarted","Data":"a658fd3e2de65460c1ec464b1719d4187800a4d0e03f306bdf9d6c7973b2cfc3"}
Oct 30 19:55:04.244977 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:04.244940    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc" event={"ID":"db8bc9fc-79eb-466f-9f07-35f37e2798d4","Type":"ContainerStarted","Data":"dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235"}
Oct 30 19:55:04.245381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:04.245012    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:04.375891 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:04.375844    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc" podStartSLOduration=2.4890764 podStartE2EDuration="3.375829424s" totalImagesPullingTime="886.753024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:04.373772947 +0000 UTC m=+2351.584990094" watchObservedRunningTime="2026-10-30 19:55:04.375829424 +0000 UTC m=+2351.587046573"
Oct 30 19:55:05.197978 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:55:05.197941    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:55:06.273793 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:06.273762    4420 generic.go:356] "Generic (PLEG): container finished" podID="773cf534-eeb3-43e2-8616-264970478e1d" containerID="80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f" exitCode=0
Oct 30 19:55:06.274289 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:06.273861    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" event={"ID":"773cf534-eeb3-43e2-8616-264970478e1d","Type":"ContainerDied","Data":"80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f"}
Oct 30 19:55:06.274289 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:06.274250    4420 scope.go:122] "RemoveContainer" containerID="80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f"
Oct 30 19:55:06.275767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:06.275737    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:55:06.275847 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:06.275805    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:55:06.275886 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:06.275855    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:55:07.291542 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:07.291506    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" event={"ID":"773cf534-eeb3-43e2-8616-264970478e1d","Type":"ContainerStarted","Data":"2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd"}
Oct 30 19:55:08.106492 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:08.106460    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:55:08.106671 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:08.106639    4420 patch_prober.go:29] interesting pod/e2e-api-b-1c5537-openshift-default-5df844678c-z827l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:55:08.106718 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:08.106683    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:55:09.106985 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:09.106951    4420 patch_prober.go:29] interesting pod/e2e-api-b-1c5537-openshift-default-5df844678c-z827l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:55:09.107377 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:09.107015    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:55:10.106932 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.106900    4420 patch_prober.go:29] interesting pod/e2e-api-b-1c5537-openshift-default-5df844678c-z827l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:55:10.107110 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.106965    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:55:10.256607 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.256580    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:55:10.299860 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.299828    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"]
Oct 30 19:55:10.300090 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.300030    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh" podUID="3d928ada-37ad-4ac0-baf0-ad3494327600" containerName="discovery" containerID="cri-o://a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3" gracePeriod=30
Oct 30 19:55:10.640795 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.640773    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:55:10.788895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.785316    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-token\" (UniqueName: \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-token\") pod \"3d928ada-37ad-4ac0-baf0-ad3494327600\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") "
Oct 30 19:55:10.788895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.785441    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-cacerts\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-cacerts\") pod \"3d928ada-37ad-4ac0-baf0-ad3494327600\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") "
Oct 30 19:55:10.788895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.785496    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-kube-api-access-cqpqh\" (UniqueName: \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-kube-api-access-cqpqh\") pod \"3d928ada-37ad-4ac0-baf0-ad3494327600\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") "
Oct 30 19:55:10.788895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.785534    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-kubeconfig\") pod \"3d928ada-37ad-4ac0-baf0-ad3494327600\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") "
Oct 30 19:55:10.788895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.785610    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-dns-cert\") pod \"3d928ada-37ad-4ac0-baf0-ad3494327600\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") "
Oct 30 19:55:10.788895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.785707    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d928ada-37ad-4ac0-baf0-ad3494327600-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d928ada-37ad-4ac0-baf0-ad3494327600-local-certs\") pod \"3d928ada-37ad-4ac0-baf0-ad3494327600\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") "
Oct 30 19:55:10.788895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.785829    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-ca-configmap\") pod \"3d928ada-37ad-4ac0-baf0-ad3494327600\" (UID: \"3d928ada-37ad-4ac0-baf0-ad3494327600\") "
Oct 30 19:55:10.789397 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.789078    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-ca-configmap" pod "3d928ada-37ad-4ac0-baf0-ad3494327600" (UID: "3d928ada-37ad-4ac0-baf0-ad3494327600"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:10.794495 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.794453    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-dns-cert" pod "3d928ada-37ad-4ac0-baf0-ad3494327600" (UID: "3d928ada-37ad-4ac0-baf0-ad3494327600"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:10.794660 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.794497    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-cacerts" pod "3d928ada-37ad-4ac0-baf0-ad3494327600" (UID: "3d928ada-37ad-4ac0-baf0-ad3494327600"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:10.794879 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.794781    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-kubeconfig" pod "3d928ada-37ad-4ac0-baf0-ad3494327600" (UID: "3d928ada-37ad-4ac0-baf0-ad3494327600"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:10.795014 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.794952    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-token" pod "3d928ada-37ad-4ac0-baf0-ad3494327600" (UID: "3d928ada-37ad-4ac0-baf0-ad3494327600"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:10.795142 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.795042    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-kube-api-access-cqpqh" pod "3d928ada-37ad-4ac0-baf0-ad3494327600" (UID: "3d928ada-37ad-4ac0-baf0-ad3494327600"). InnerVolumeSpecName "kube-api-access-cqpqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:10.796122 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.796089    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d928ada-37ad-4ac0-baf0-ad3494327600-local-certs" pod "3d928ada-37ad-4ac0-baf0-ad3494327600" (UID: "3d928ada-37ad-4ac0-baf0-ad3494327600"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:10.892563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.892429    4420 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-ca-configmap\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:10.892563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.892468    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:10.892563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.892484    4420 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-cacerts\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:10.892563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.892498    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqpqh\" (UniqueName: \"kubernetes.io/projected/3d928ada-37ad-4ac0-baf0-ad3494327600-kube-api-access-cqpqh\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:10.892563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.892512    4420 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-kubeconfig\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:10.892563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.892525    4420 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d928ada-37ad-4ac0-baf0-ad3494327600-istio-csr-dns-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:10.892563 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:10.892538    4420 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d928ada-37ad-4ac0-baf0-ad3494327600-local-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:11.106972 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.106883    4420 patch_prober.go:29] interesting pod/e2e-api-b-1c5537-openshift-default-5df844678c-z827l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:55:11.106972 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.106941    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:55:11.144723 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.144694    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"]
Oct 30 19:55:11.145426 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.145407    4420 cpu_manager.go:414] "containerMap: removing container" podUID="3d928ada-37ad-4ac0-baf0-ad3494327600" containerName="discovery"
Oct 30 19:55:11.145524 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.145427    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d928ada-37ad-4ac0-baf0-ad3494327600" containerName="discovery"
Oct 30 19:55:11.145603 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.145589    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d928ada-37ad-4ac0-baf0-ad3494327600" containerName="discovery"
Oct 30 19:55:11.149479 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.149460    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.201678 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.201651    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6caff5-serving-cert\""
Oct 30 19:55:11.214091 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.214059    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"]
Oct 30 19:55:11.298209 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.298175    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-openshift-service-ca\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.298378 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.298295    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9vh\" (UniqueName: \"kubernetes.io/projected/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-kube-api-access-2m9vh\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.298378 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.298364    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-maas-api-tls\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.298577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.298557    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-metrics-certs\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.325180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.321167    4420 generic.go:356] "Generic (PLEG): container finished" podID="3d928ada-37ad-4ac0-baf0-ad3494327600" containerID="a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3" exitCode=0
Oct 30 19:55:11.325180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.321255    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh" event={"ID":"3d928ada-37ad-4ac0-baf0-ad3494327600","Type":"ContainerDied","Data":"a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3"}
Oct 30 19:55:11.325180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.321299    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh" event={"ID":"3d928ada-37ad-4ac0-baf0-ad3494327600","Type":"ContainerDied","Data":"31bc581177f1987257516ce033055db9994a6b1f6c43f9f9480e23f919c14a2f"}
Oct 30 19:55:11.325180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.321323    4420 scope.go:122] "RemoveContainer" containerID="a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3"
Oct 30 19:55:11.325180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.321482    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"
Oct 30 19:55:11.333778 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.333744    4420 scope.go:122] "RemoveContainer" containerID="a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3"
Oct 30 19:55:11.334048 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:55:11.334028    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3\": container with ID starting with a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3 not found: ID does not exist" containerID="a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3"
Oct 30 19:55:11.334117 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.334061    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3"} err="failed to get container status \"a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3\": rpc error: code = NotFound desc = could not find container \"a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3\": container with ID starting with a077a3d08fb24053a26bf885aa7e6d8f13777372b50017affcc713f04e5231a3 not found: ID does not exist"
Oct 30 19:55:11.400480 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.400450    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-openshift-service-ca\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.400624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.400512    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9vh\" (UniqueName: \"kubernetes.io/projected/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-kube-api-access-2m9vh\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.400624 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.400541    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-maas-api-tls\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.400713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.400645    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-metrics-certs\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.401198 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.401167    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-openshift-service-ca\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.403175 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.403153    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-metrics-certs\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.403285 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.403217    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-maas-api-tls\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.456763 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.456733    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9vh\" (UniqueName: \"kubernetes.io/projected/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-kube-api-access-2m9vh\") pod \"maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.482836 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.479251    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:11.494052 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.493840    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"]
Oct 30 19:55:11.499176 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.499148    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kqkhh"]
Oct 30 19:55:11.885702 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:11.885670    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"]
Oct 30 19:55:11.886231 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:55:11.886206    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cb0837_aeb9_45d0_9f11_aa55e9b2a124.slice/crio-3b4e627f77aa3b9ee40cb22d1f0144278c986e00325ad28d5a3db4fd3362aab8 WatchSource:0}: Error finding container 3b4e627f77aa3b9ee40cb22d1f0144278c986e00325ad28d5a3db4fd3362aab8: Status 404 returned error can't find the container with id 3b4e627f77aa3b9ee40cb22d1f0144278c986e00325ad28d5a3db4fd3362aab8
Oct 30 19:55:12.107164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:12.107129    4420 patch_prober.go:29] interesting pod/e2e-api-b-1c5537-openshift-default-5df844678c-z827l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:55:12.107555 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:12.107179    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 19:55:12.329175 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:12.329139    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m" event={"ID":"44cb0837-aeb9-45d0-9f11-aa55e9b2a124","Type":"ContainerStarted","Data":"3b4e627f77aa3b9ee40cb22d1f0144278c986e00325ad28d5a3db4fd3362aab8"}
Oct 30 19:55:12.899114 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:12.899078    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d928ada-37ad-4ac0-baf0-ad3494327600" path="/var/lib/kubelet/pods/3d928ada-37ad-4ac0-baf0-ad3494327600/volumes"
Oct 30 19:55:13.111502 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:13.111425    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:55:13.111885 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:13.111835    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:55:13.112492 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:13.112473    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 19:55:13.338666 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:13.338628    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m" event={"ID":"44cb0837-aeb9-45d0-9f11-aa55e9b2a124","Type":"ContainerStarted","Data":"00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b"}
Oct 30 19:55:13.338848 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:13.338711    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:13.555885 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:13.555802    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m" podStartSLOduration=1.6676597960000001 podStartE2EDuration="2.555788104s" totalImagesPullingTime="888.128308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:13.553991335 +0000 UTC m=+2360.765208478" watchObservedRunningTime="2026-10-30 19:55:13.555788104 +0000 UTC m=+2360.767005251"
Oct 30 19:55:19.351448 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:19.351422    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:55:29.741092 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:29.741063    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"]
Oct 30 19:55:29.741545 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:29.741265    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9" podUID="3b427057-1f04-4da3-a6d7-2d072caf1e12" containerName="maas-api" containerID="cri-o://457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c" gracePeriod=30
Oct 30 19:55:30.096122 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.096084    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:55:30.207912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.207885    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b427057-1f04-4da3-a6d7-2d072caf1e12-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b427057-1f04-4da3-a6d7-2d072caf1e12-openshift-service-ca\") pod \"3b427057-1f04-4da3-a6d7-2d072caf1e12\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") "
Oct 30 19:55:30.208102 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.207921    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-maas-api-tls\") pod \"3b427057-1f04-4da3-a6d7-2d072caf1e12\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") "
Oct 30 19:55:30.208102 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.208008    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b427057-1f04-4da3-a6d7-2d072caf1e12-kube-api-access-sxr9d\" (UniqueName: \"kubernetes.io/projected/3b427057-1f04-4da3-a6d7-2d072caf1e12-kube-api-access-sxr9d\") pod \"3b427057-1f04-4da3-a6d7-2d072caf1e12\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") "
Oct 30 19:55:30.208257 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.208237    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-metrics-certs\") pod \"3b427057-1f04-4da3-a6d7-2d072caf1e12\" (UID: \"3b427057-1f04-4da3-a6d7-2d072caf1e12\") "
Oct 30 19:55:30.208324 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.208264    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b427057-1f04-4da3-a6d7-2d072caf1e12-openshift-service-ca" pod "3b427057-1f04-4da3-a6d7-2d072caf1e12" (UID: "3b427057-1f04-4da3-a6d7-2d072caf1e12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:30.208763 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.208739    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b427057-1f04-4da3-a6d7-2d072caf1e12-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.210247 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.210214    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-maas-api-tls" pod "3b427057-1f04-4da3-a6d7-2d072caf1e12" (UID: "3b427057-1f04-4da3-a6d7-2d072caf1e12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:30.210337 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.210259    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-metrics-certs" pod "3b427057-1f04-4da3-a6d7-2d072caf1e12" (UID: "3b427057-1f04-4da3-a6d7-2d072caf1e12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:55:30.210386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.210355    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b427057-1f04-4da3-a6d7-2d072caf1e12-kube-api-access-sxr9d" pod "3b427057-1f04-4da3-a6d7-2d072caf1e12" (UID: "3b427057-1f04-4da3-a6d7-2d072caf1e12"). InnerVolumeSpecName "kube-api-access-sxr9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:30.313249 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.313127    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.313488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.313472    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b427057-1f04-4da3-a6d7-2d072caf1e12-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.313573 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.313563    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxr9d\" (UniqueName: \"kubernetes.io/projected/3b427057-1f04-4da3-a6d7-2d072caf1e12-kube-api-access-sxr9d\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:30.452120 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.452086    4420 generic.go:356] "Generic (PLEG): container finished" podID="3b427057-1f04-4da3-a6d7-2d072caf1e12" containerID="457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c" exitCode=0
Oct 30 19:55:30.452293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.452138    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"
Oct 30 19:55:30.452293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.452160    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9" event={"ID":"3b427057-1f04-4da3-a6d7-2d072caf1e12","Type":"ContainerDied","Data":"457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c"}
Oct 30 19:55:30.452293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.452218    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9" event={"ID":"3b427057-1f04-4da3-a6d7-2d072caf1e12","Type":"ContainerDied","Data":"e234c0674882c7e8421f03fd3a22ef4001a88b3ebb6659b47f13d0c25c4b90c5"}
Oct 30 19:55:30.452293 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.452241    4420 scope.go:122] "RemoveContainer" containerID="457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c"
Oct 30 19:55:30.472277 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.472253    4420 scope.go:122] "RemoveContainer" containerID="457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c"
Oct 30 19:55:30.472672 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:55:30.472642    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c\": container with ID starting with 457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c not found: ID does not exist" containerID="457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c"
Oct 30 19:55:30.472781 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.472681    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c"} err="failed to get container status \"457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c\": rpc error: code = NotFound desc = could not find container \"457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c\": container with ID starting with 457bc3a0c2b505e2115cb29ed763d30ee2c09b5108a5a2812ab2a1e81e1b693c not found: ID does not exist"
Oct 30 19:55:30.617022 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.616943    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"]
Oct 30 19:55:30.620072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.620053    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-bfdf9"]
Oct 30 19:55:30.897037 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:30.896996    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b427057-1f04-4da3-a6d7-2d072caf1e12" path="/var/lib/kubelet/pods/3b427057-1f04-4da3-a6d7-2d072caf1e12/volumes"
Oct 30 19:55:35.198514 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:55:35.198483    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:55:35.474469 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:35.474392    4420 patch_prober.go:29] interesting pod/e2e-api-b-da4c62-openshift-default-5945967dc-zls87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:55:35.474469 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:35.474456    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 19:55:35.513263 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:35.513237    4420 generic.go:356] "Generic (PLEG): container finished" podID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerID="7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe" exitCode=0
Oct 30 19:55:35.513396 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:35.513307    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" event={"ID":"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e","Type":"ContainerDied","Data":"7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe"}
Oct 30 19:55:35.513745 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:35.513729    4420 scope.go:122] "RemoveContainer" containerID="7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe"
Oct 30 19:55:35.515454 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:35.515427    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:55:35.515538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:35.515496    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:55:35.515538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:35.515529    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:55:36.524000 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:36.523965    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" event={"ID":"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e","Type":"ContainerStarted","Data":"d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1"}
Oct 30 19:55:37.127171 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:37.127144    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:55:37.127374 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:37.127268    4420 patch_prober.go:29] interesting pod/e2e-api-b-da4c62-openshift-default-5945967dc-zls87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:55:37.127374 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:37.127319    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 19:55:38.127779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:38.127743    4420 patch_prober.go:29] interesting pod/e2e-api-b-da4c62-openshift-default-5945967dc-zls87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:55:38.128234 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:38.127808    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 19:55:39.127749 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:39.127709    4420 patch_prober.go:29] interesting pod/e2e-api-b-da4c62-openshift-default-5945967dc-zls87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:55:39.127967 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:39.127790    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 19:55:40.127332 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:40.127297    4420 patch_prober.go:29] interesting pod/e2e-api-b-da4c62-openshift-default-5945967dc-zls87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:55:40.127523 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:40.127359    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 19:55:41.128209 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:41.128170    4420 patch_prober.go:29] interesting pod/e2e-api-b-da4c62-openshift-default-5945967dc-zls87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:55:41.128594 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:41.128242    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 30 19:55:42.133744 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:42.133712    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:55:42.134278 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:42.134167    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:55:42.134932 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:42.134911    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:55:42.433409 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:42.433231    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"]
Oct 30 19:55:42.438462 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:42.438422    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy" containerID="cri-o://fc073dc3f7add73f1dca686f0313b54211ff32fb898df30548b743ddd318e4fe" gracePeriod=30
Oct 30 19:55:43.115148 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:43.115117    4420 patch_prober.go:29] interesting pod/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:55:43.115342 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:43.115169    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:55:47.628029 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:47.627939    4420 generic.go:356] "Generic (PLEG): container finished" podID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerID="fc073dc3f7add73f1dca686f0313b54211ff32fb898df30548b743ddd318e4fe" exitCode=0
Oct 30 19:55:47.628029 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:47.627998    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" event={"ID":"a7f4fdbb-d865-42f6-92cd-331a5468deed","Type":"ContainerDied","Data":"fc073dc3f7add73f1dca686f0313b54211ff32fb898df30548b743ddd318e4fe"}
Oct 30 19:55:47.628520 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:47.628068    4420 scope.go:122] "RemoveContainer" containerID="60e21861f120f21af8df152a3500cffd24814e8182ebadcc1056a106d838e143"
Oct 30 19:55:48.120920 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.120897    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:55:48.283751 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.283682    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-kube-api-access-phg68\" (UniqueName: \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-kube-api-access-phg68\") pod \"a7f4fdbb-d865-42f6-92cd-331a5468deed\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") "
Oct 30 19:55:48.283939 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.283758    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-credential-socket\") pod \"a7f4fdbb-d865-42f6-92cd-331a5468deed\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") "
Oct 30 19:55:48.283939 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.283786    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-envoy\") pod \"a7f4fdbb-d865-42f6-92cd-331a5468deed\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") "
Oct 30 19:55:48.283939 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.283836    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-podinfo\") pod \"a7f4fdbb-d865-42f6-92cd-331a5468deed\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") "
Oct 30 19:55:48.283939 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.283866    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-token\" (UniqueName: \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-token\") pod \"a7f4fdbb-d865-42f6-92cd-331a5468deed\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") "
Oct 30 19:55:48.284183 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.283945    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-certs\") pod \"a7f4fdbb-d865-42f6-92cd-331a5468deed\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") "
Oct 30 19:55:48.284183 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.284007    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f4fdbb-d865-42f6-92cd-331a5468deed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f4fdbb-d865-42f6-92cd-331a5468deed-istiod-ca-cert\") pod \"a7f4fdbb-d865-42f6-92cd-331a5468deed\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") "
Oct 30 19:55:48.284183 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.284055    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-credential-socket" pod "a7f4fdbb-d865-42f6-92cd-331a5468deed" (UID: "a7f4fdbb-d865-42f6-92cd-331a5468deed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.284183 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.284071    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-socket\") pod \"a7f4fdbb-d865-42f6-92cd-331a5468deed\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") "
Oct 30 19:55:48.284183 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.284145    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-data\") pod \"a7f4fdbb-d865-42f6-92cd-331a5468deed\" (UID: \"a7f4fdbb-d865-42f6-92cd-331a5468deed\") "
Oct 30 19:55:48.284460 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.284281    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-socket" pod "a7f4fdbb-d865-42f6-92cd-331a5468deed" (UID: "a7f4fdbb-d865-42f6-92cd-331a5468deed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.284460 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.284286    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-certs" pod "a7f4fdbb-d865-42f6-92cd-331a5468deed" (UID: "a7f4fdbb-d865-42f6-92cd-331a5468deed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.284730 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.284564    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-data" pod "a7f4fdbb-d865-42f6-92cd-331a5468deed" (UID: "a7f4fdbb-d865-42f6-92cd-331a5468deed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.285143 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.285116    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.285143 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.285143    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.285330 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.285177    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.285330 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.285192    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.285330 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.285217    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f4fdbb-d865-42f6-92cd-331a5468deed-istiod-ca-cert" pod "a7f4fdbb-d865-42f6-92cd-331a5468deed" (UID: "a7f4fdbb-d865-42f6-92cd-331a5468deed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:55:48.286391 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.286345    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-kube-api-access-phg68" pod "a7f4fdbb-d865-42f6-92cd-331a5468deed" (UID: "a7f4fdbb-d865-42f6-92cd-331a5468deed"). InnerVolumeSpecName "kube-api-access-phg68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:48.286622 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.286569    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-podinfo" pod "a7f4fdbb-d865-42f6-92cd-331a5468deed" (UID: "a7f4fdbb-d865-42f6-92cd-331a5468deed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:55:48.287164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.287131    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-envoy" pod "a7f4fdbb-d865-42f6-92cd-331a5468deed" (UID: "a7f4fdbb-d865-42f6-92cd-331a5468deed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:55:48.288168 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.288142    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-token" pod "a7f4fdbb-d865-42f6-92cd-331a5468deed" (UID: "a7f4fdbb-d865-42f6-92cd-331a5468deed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:55:48.386103 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.386068    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phg68\" (UniqueName: \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-kube-api-access-phg68\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.386103 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.386100    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.386300 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.386114    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.386300 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.386126    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7f4fdbb-d865-42f6-92cd-331a5468deed-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.386300 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.386138    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7f4fdbb-d865-42f6-92cd-331a5468deed-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:55:48.636514 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.636437    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz" event={"ID":"a7f4fdbb-d865-42f6-92cd-331a5468deed","Type":"ContainerDied","Data":"af7b073d8ece474ee55166bae88cfab0b2bc8ffd530617c85036c12d57c4f1a3"}
Oct 30 19:55:48.636972 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.636525    4420 scope.go:122] "RemoveContainer" containerID="fc073dc3f7add73f1dca686f0313b54211ff32fb898df30548b743ddd318e4fe"
Oct 30 19:55:48.636972 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.636445    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"
Oct 30 19:55:48.801288 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.801263    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"]
Oct 30 19:55:48.850740 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.850710    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ada8d7-openshift-default-5f8b49bdfc-gt8lz"]
Oct 30 19:55:48.896663 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:48.896638    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" path="/var/lib/kubelet/pods/a7f4fdbb-d865-42f6-92cd-331a5468deed/volumes"
Oct 30 19:55:51.021092 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.021065    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"]
Oct 30 19:55:51.021684 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.021672    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy"
Oct 30 19:55:51.021738 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.021685    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy"
Oct 30 19:55:51.021738 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.021710    4420 cpu_manager.go:414] "containerMap: removing container" podUID="3b427057-1f04-4da3-a6d7-2d072caf1e12" containerName="maas-api"
Oct 30 19:55:51.021738 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.021715    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b427057-1f04-4da3-a6d7-2d072caf1e12" containerName="maas-api"
Oct 30 19:55:51.021866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.021838    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b427057-1f04-4da3-a6d7-2d072caf1e12" containerName="maas-api"
Oct 30 19:55:51.021866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.021852    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy"
Oct 30 19:55:51.021866 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.021861    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy"
Oct 30 19:55:51.024884 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.024870    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.079961 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.079933    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-75b38d-serving-cert\""
Oct 30 19:55:51.118108 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.118085    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-metrics-certs\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.118255 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.118117    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-maas-api-tls\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.118255 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.118241    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxrhk\" (UniqueName: \"kubernetes.io/projected/089b836d-7fcf-4210-9259-856c8d8fde41-kube-api-access-hxrhk\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.118428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.118409    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/089b836d-7fcf-4210-9259-856c8d8fde41-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.133104 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.133076    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"]
Oct 30 19:55:51.219436 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.219397    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/089b836d-7fcf-4210-9259-856c8d8fde41-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.219611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.219482    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-metrics-certs\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.219611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.219512    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-maas-api-tls\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.219611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.219596    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrhk\" (UniqueName: \"kubernetes.io/projected/089b836d-7fcf-4210-9259-856c8d8fde41-kube-api-access-hxrhk\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.219772 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:55:51.219702    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-serving-cert: secret "maas-api-e2e-inf-b-75b38d-serving-cert" not found
Oct 30 19:55:51.219835 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:55:51.219779    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-maas-api-tls podName:089b836d-7fcf-4210-9259-856c8d8fde41 nodeName:}" failed. No retries permitted until 2026-10-30 19:55:51.719757534 +0000 UTC m=+2398.930974678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-maas-api-tls") pod "maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w" (UID: "089b836d-7fcf-4210-9259-856c8d8fde41") : secret "maas-api-e2e-inf-b-75b38d-serving-cert" not found
Oct 30 19:55:51.220164 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.220144    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/089b836d-7fcf-4210-9259-856c8d8fde41-openshift-service-ca\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.222003 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.221984    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-metrics-certs\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.279340 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.278856    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrhk\" (UniqueName: \"kubernetes.io/projected/089b836d-7fcf-4210-9259-856c8d8fde41-kube-api-access-hxrhk\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.727073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.727045    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-maas-api-tls\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.729399 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.729372    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-maas-api-tls\") pod \"maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:51.937975 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:51.937947    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:52.312501 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:52.312473    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"]
Oct 30 19:55:52.313072 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:55:52.313048    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089b836d_7fcf_4210_9259_856c8d8fde41.slice/crio-fbd67e64143ede7e19cf1d2a928820d6d7947051443c2855e1eb1311f00874d6 WatchSource:0}: Error finding container fbd67e64143ede7e19cf1d2a928820d6d7947051443c2855e1eb1311f00874d6: Status 404 returned error can't find the container with id fbd67e64143ede7e19cf1d2a928820d6d7947051443c2855e1eb1311f00874d6
Oct 30 19:55:52.665799 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:52.665766    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w" event={"ID":"089b836d-7fcf-4210-9259-856c8d8fde41","Type":"ContainerStarted","Data":"fbd67e64143ede7e19cf1d2a928820d6d7947051443c2855e1eb1311f00874d6"}
Oct 30 19:55:53.673258 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:53.673228    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w" event={"ID":"089b836d-7fcf-4210-9259-856c8d8fde41","Type":"ContainerStarted","Data":"2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560"}
Oct 30 19:55:53.673615 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:53.673405    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:55:53.789419 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:53.789317    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w" podStartSLOduration=2.975726994 podStartE2EDuration="3.789302175s" totalImagesPullingTime="813.575181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:53.787751763 +0000 UTC m=+2400.998968909" watchObservedRunningTime="2026-10-30 19:55:53.789302175 +0000 UTC m=+2401.000519357"
Oct 30 19:55:59.684940 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:55:59.684910    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:56:05.198980 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:56:05.198939    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:56:35.197711 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:56:35.197676    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:56:42.065714 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:42.065679    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"]
Oct 30 19:56:47.066924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:47.066858    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp" podUID="39ab0175-196a-4ea7-be0a-69d8973befd2" containerName="payload-processing" containerID="cri-o://3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89" gracePeriod=25
Oct 30 19:56:47.120184 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:56:47.120164    4420 logging.go:55] [core] [Channel #572 SubChannel #573]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.93:50052", ServerName: "10.133.0.93:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.93:50052: connect: connection refused"
Oct 30 19:56:47.343276 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:47.343249    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:56:47.407892 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:47.407870    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ab0175-196a-4ea7-be0a-69d8973befd2-kube-api-access-725k4\" (UniqueName: \"kubernetes.io/projected/39ab0175-196a-4ea7-be0a-69d8973befd2-kube-api-access-725k4\") pod \"39ab0175-196a-4ea7-be0a-69d8973befd2\" (UID: \"39ab0175-196a-4ea7-be0a-69d8973befd2\") "
Oct 30 19:56:47.408015 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:47.407938    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ab0175-196a-4ea7-be0a-69d8973befd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ab0175-196a-4ea7-be0a-69d8973befd2-plugins-config-volume\") pod \"39ab0175-196a-4ea7-be0a-69d8973befd2\" (UID: \"39ab0175-196a-4ea7-be0a-69d8973befd2\") "
Oct 30 19:56:47.408320 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:47.408294    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ab0175-196a-4ea7-be0a-69d8973befd2-plugins-config-volume" pod "39ab0175-196a-4ea7-be0a-69d8973befd2" (UID: "39ab0175-196a-4ea7-be0a-69d8973befd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:56:47.408469 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:47.408455    4420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39ab0175-196a-4ea7-be0a-69d8973befd2-plugins-config-volume\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:47.410170 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:47.410144    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ab0175-196a-4ea7-be0a-69d8973befd2-kube-api-access-725k4" pod "39ab0175-196a-4ea7-be0a-69d8973befd2" (UID: "39ab0175-196a-4ea7-be0a-69d8973befd2"). InnerVolumeSpecName "kube-api-access-725k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:56:47.510169 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:47.510133    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-725k4\" (UniqueName: \"kubernetes.io/projected/39ab0175-196a-4ea7-be0a-69d8973befd2-kube-api-access-725k4\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:56:48.105283 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:48.105249    4420 generic.go:356] "Generic (PLEG): container finished" podID="39ab0175-196a-4ea7-be0a-69d8973befd2" containerID="3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89" exitCode=0
Oct 30 19:56:48.105687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:48.105297    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"
Oct 30 19:56:48.105687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:48.105321    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp" event={"ID":"39ab0175-196a-4ea7-be0a-69d8973befd2","Type":"ContainerDied","Data":"3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89"}
Oct 30 19:56:48.105687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:48.105373    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp" event={"ID":"39ab0175-196a-4ea7-be0a-69d8973befd2","Type":"ContainerDied","Data":"6c61abd84ae3ce62e45995246e954fc1730ab2d942ced72e9f2ddf5cef7b84fc"}
Oct 30 19:56:48.105687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:48.105402    4420 scope.go:122] "RemoveContainer" containerID="3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89"
Oct 30 19:56:48.115888 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:48.115871    4420 scope.go:122] "RemoveContainer" containerID="3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89"
Oct 30 19:56:48.116144 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:56:48.116122    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89\": container with ID starting with 3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89 not found: ID does not exist" containerID="3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89"
Oct 30 19:56:48.116196 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:48.116152    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89"} err="failed to get container status \"3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89\": rpc error: code = NotFound desc = could not find container \"3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89\": container with ID starting with 3792d8fd6f9d489d570d96bff33170aacada73b95adc56b0f14e102e8e460a89 not found: ID does not exist"
Oct 30 19:56:48.120708 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:48.120661    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp" podUID="39ab0175-196a-4ea7-be0a-69d8973befd2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.93:50052\" within 1s: context deadline exceeded"
Oct 30 19:56:48.268574 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:48.268550    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"]
Oct 30 19:56:48.272269 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:48.272238    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-daa1ac-dd7ccdfc8-rkmxp"]
Oct 30 19:56:48.897381 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:56:48.897353    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ab0175-196a-4ea7-be0a-69d8973befd2" path="/var/lib/kubelet/pods/39ab0175-196a-4ea7-be0a-69d8973befd2/volumes"
Oct 30 19:57:00.211585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.211552    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"]
Oct 30 19:57:00.212036 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.211751    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m" podUID="44cb0837-aeb9-45d0-9f11-aa55e9b2a124" containerName="maas-api" containerID="cri-o://00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b" gracePeriod=30
Oct 30 19:57:00.517484 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.517457    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:57:00.664405 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.664376    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-openshift-service-ca\") pod \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") "
Oct 30 19:57:00.664707 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.664684    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-metrics-certs\") pod \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") "
Oct 30 19:57:00.664852 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.664754    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-openshift-service-ca" pod "44cb0837-aeb9-45d0-9f11-aa55e9b2a124" (UID: "44cb0837-aeb9-45d0-9f11-aa55e9b2a124"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:57:00.664910 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.664852    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-kube-api-access-2m9vh\" (UniqueName: \"kubernetes.io/projected/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-kube-api-access-2m9vh\") pod \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") "
Oct 30 19:57:00.664910 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.664883    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-maas-api-tls\") pod \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\" (UID: \"44cb0837-aeb9-45d0-9f11-aa55e9b2a124\") "
Oct 30 19:57:00.665590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.665525    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:00.667233 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.667208    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-kube-api-access-2m9vh" pod "44cb0837-aeb9-45d0-9f11-aa55e9b2a124" (UID: "44cb0837-aeb9-45d0-9f11-aa55e9b2a124"). InnerVolumeSpecName "kube-api-access-2m9vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:57:00.669307 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.669282    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-maas-api-tls" pod "44cb0837-aeb9-45d0-9f11-aa55e9b2a124" (UID: "44cb0837-aeb9-45d0-9f11-aa55e9b2a124"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:00.670429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.670411    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-metrics-certs" pod "44cb0837-aeb9-45d0-9f11-aa55e9b2a124" (UID: "44cb0837-aeb9-45d0-9f11-aa55e9b2a124"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:57:00.767591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.767443    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:00.767591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.767484    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9vh\" (UniqueName: \"kubernetes.io/projected/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-kube-api-access-2m9vh\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:00.767591 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:00.767501    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44cb0837-aeb9-45d0-9f11-aa55e9b2a124-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:57:01.195201 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:01.195167    4420 generic.go:356] "Generic (PLEG): container finished" podID="44cb0837-aeb9-45d0-9f11-aa55e9b2a124" containerID="00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b" exitCode=0
Oct 30 19:57:01.195377 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:01.195214    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"
Oct 30 19:57:01.195377 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:01.195250    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m" event={"ID":"44cb0837-aeb9-45d0-9f11-aa55e9b2a124","Type":"ContainerDied","Data":"00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b"}
Oct 30 19:57:01.195377 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:01.195311    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m" event={"ID":"44cb0837-aeb9-45d0-9f11-aa55e9b2a124","Type":"ContainerDied","Data":"3b4e627f77aa3b9ee40cb22d1f0144278c986e00325ad28d5a3db4fd3362aab8"}
Oct 30 19:57:01.195377 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:01.195337    4420 scope.go:122] "RemoveContainer" containerID="00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b"
Oct 30 19:57:01.205488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:01.205457    4420 scope.go:122] "RemoveContainer" containerID="00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b"
Oct 30 19:57:01.205740 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:57:01.205718    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b\": container with ID starting with 00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b not found: ID does not exist" containerID="00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b"
Oct 30 19:57:01.205830 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:01.205751    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b"} err="failed to get container status \"00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b\": rpc error: code = NotFound desc = could not find container \"00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b\": container with ID starting with 00bcbab2765413b70d8e322b5cd6e828d8256d847cf0729f791bce23080b126b not found: ID does not exist"
Oct 30 19:57:01.359429 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:01.359405    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"]
Oct 30 19:57:01.409119 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:01.409085    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6caff5-5b84f5f4bf-lsq5m"]
Oct 30 19:57:02.896581 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:02.896548    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44cb0837-aeb9-45d0-9f11-aa55e9b2a124" path="/var/lib/kubelet/pods/44cb0837-aeb9-45d0-9f11-aa55e9b2a124/volumes"
Oct 30 19:57:05.197052 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:57:05.196980    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:57:16.949568 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:16.949540    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"]
Oct 30 19:57:16.950222 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:16.950207    4420 cpu_manager.go:414] "containerMap: removing container" podUID="39ab0175-196a-4ea7-be0a-69d8973befd2" containerName="payload-processing"
Oct 30 19:57:16.950222 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:16.950223    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ab0175-196a-4ea7-be0a-69d8973befd2" containerName="payload-processing"
Oct 30 19:57:16.950328 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:16.950252    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy"
Oct 30 19:57:16.950328 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:16.950257    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f4fdbb-d865-42f6-92cd-331a5468deed" containerName="istio-proxy"
Oct 30 19:57:16.950328 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:16.950291    4420 cpu_manager.go:414] "containerMap: removing container" podUID="44cb0837-aeb9-45d0-9f11-aa55e9b2a124" containerName="maas-api"
Oct 30 19:57:16.950328 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:16.950295    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44cb0837-aeb9-45d0-9f11-aa55e9b2a124" containerName="maas-api"
Oct 30 19:57:16.950443 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:16.950406    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="44cb0837-aeb9-45d0-9f11-aa55e9b2a124" containerName="maas-api"
Oct 30 19:57:16.950443 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:16.950417    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ab0175-196a-4ea7-be0a-69d8973befd2" containerName="payload-processing"
Oct 30 19:57:16.955364 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:16.955347    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.009199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.009176    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-dd9b8d79-serving-cert\""
Oct 30 19:57:17.062208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.062178    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"]
Oct 30 19:57:17.066658 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.066631    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c081125-76ff-454c-80f5-2f06fe73dbd8-openshift-service-ca\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.066772 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.066691    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-metrics-certs\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.066831 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.066782    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-maas-api-tls\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.066913 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.066894    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d8rk\" (UniqueName: \"kubernetes.io/projected/2c081125-76ff-454c-80f5-2f06fe73dbd8-kube-api-access-2d8rk\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.167669 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.167629    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8rk\" (UniqueName: \"kubernetes.io/projected/2c081125-76ff-454c-80f5-2f06fe73dbd8-kube-api-access-2d8rk\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.167877 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.167860    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c081125-76ff-454c-80f5-2f06fe73dbd8-openshift-service-ca\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.167968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.167931    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-metrics-certs\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.168028 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.167996    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-maas-api-tls\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.168176 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:57:17.168159    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-serving-cert: secret "maas-api-e2e-mt-dd9b8d79-serving-cert" not found
Oct 30 19:57:17.168253 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:57:17.168240    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-maas-api-tls podName:2c081125-76ff-454c-80f5-2f06fe73dbd8 nodeName:}" failed. No retries permitted until 2026-10-30 19:57:17.66821997 +0000 UTC m=+2484.879437109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-maas-api-tls") pod "maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p" (UID: "2c081125-76ff-454c-80f5-2f06fe73dbd8") : secret "maas-api-e2e-mt-dd9b8d79-serving-cert" not found
Oct 30 19:57:17.168551 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.168532    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c081125-76ff-454c-80f5-2f06fe73dbd8-openshift-service-ca\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.170442 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.170424    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-metrics-certs\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.236096 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.236021    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8rk\" (UniqueName: \"kubernetes.io/projected/2c081125-76ff-454c-80f5-2f06fe73dbd8-kube-api-access-2d8rk\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.675605 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.675570    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-maas-api-tls\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.678212 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.678183    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-maas-api-tls\") pod \"maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:17.870414 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:17.870369    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:18.256450 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:18.256420    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"]
Oct 30 19:57:18.258134 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:57:18.258101    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c081125_76ff_454c_80f5_2f06fe73dbd8.slice/crio-adbcbd8dfa08594f741ba64842777b1497c927d786c2be95550df4a5dca19152 WatchSource:0}: Error finding container adbcbd8dfa08594f741ba64842777b1497c927d786c2be95550df4a5dca19152: Status 404 returned error can't find the container with id adbcbd8dfa08594f741ba64842777b1497c927d786c2be95550df4a5dca19152
Oct 30 19:57:18.260149 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:18.260129    4420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:57:18.314550 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:18.314526    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p" event={"ID":"2c081125-76ff-454c-80f5-2f06fe73dbd8","Type":"ContainerStarted","Data":"adbcbd8dfa08594f741ba64842777b1497c927d786c2be95550df4a5dca19152"}
Oct 30 19:57:19.325018 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:19.324973    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p" event={"ID":"2c081125-76ff-454c-80f5-2f06fe73dbd8","Type":"ContainerStarted","Data":"982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0"}
Oct 30 19:57:19.325516 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:19.325041    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:19.445441 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:19.445396    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p" podStartSLOduration=2.602964515 podStartE2EDuration="3.445382849s" totalImagesPullingTime="842.418334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:19.442431146 +0000 UTC m=+2486.653648290" watchObservedRunningTime="2026-10-30 19:57:19.445382849 +0000 UTC m=+2486.656599993"
Oct 30 19:57:25.336271 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:57:25.336239    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:57:35.197382 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:57:35.197342    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:58:02.831905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:02.831870    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"]
Oct 30 19:58:02.832396 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:02.832068    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh" podUID="6fa6ea6a-ca2f-49e7-9a94-75a001e56a42" containerName="maas-api" containerID="cri-o://08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7" gracePeriod=30
Oct 30 19:58:03.125587 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.125554    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:58:03.246887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.246804    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-openshift-service-ca\") pod \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") "
Oct 30 19:58:03.247104 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.246975    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-metrics-certs\") pod \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") "
Oct 30 19:58:03.247245 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.247121    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-kube-api-access-rk7pd\" (UniqueName: \"kubernetes.io/projected/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-kube-api-access-rk7pd\") pod \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") "
Oct 30 19:58:03.247245 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.247174    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-maas-api-tls\") pod \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\" (UID: \"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42\") "
Oct 30 19:58:03.248043 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.248016    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-openshift-service-ca" pod "6fa6ea6a-ca2f-49e7-9a94-75a001e56a42" (UID: "6fa6ea6a-ca2f-49e7-9a94-75a001e56a42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:03.250249 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.250221    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-metrics-certs" pod "6fa6ea6a-ca2f-49e7-9a94-75a001e56a42" (UID: "6fa6ea6a-ca2f-49e7-9a94-75a001e56a42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:03.250894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.250865    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-maas-api-tls" pod "6fa6ea6a-ca2f-49e7-9a94-75a001e56a42" (UID: "6fa6ea6a-ca2f-49e7-9a94-75a001e56a42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:03.251265 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.251241    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-kube-api-access-rk7pd" pod "6fa6ea6a-ca2f-49e7-9a94-75a001e56a42" (UID: "6fa6ea6a-ca2f-49e7-9a94-75a001e56a42"). InnerVolumeSpecName "kube-api-access-rk7pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:03.349585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.349515    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:03.349585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.349542    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:03.349585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.349556    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk7pd\" (UniqueName: \"kubernetes.io/projected/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-kube-api-access-rk7pd\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:03.349585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.349568    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:03.652779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.652750    4420 generic.go:356] "Generic (PLEG): container finished" podID="6fa6ea6a-ca2f-49e7-9a94-75a001e56a42" containerID="08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7" exitCode=0
Oct 30 19:58:03.652980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.652873    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh" event={"ID":"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42","Type":"ContainerDied","Data":"08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7"}
Oct 30 19:58:03.652980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.652912    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh" event={"ID":"6fa6ea6a-ca2f-49e7-9a94-75a001e56a42","Type":"ContainerDied","Data":"db2d0b60ed30b11866497fb2d49ffd85eae65ed3e7591b29e5a0994807166e67"}
Oct 30 19:58:03.652980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.652932    4420 scope.go:122] "RemoveContainer" containerID="08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7"
Oct 30 19:58:03.653091 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.653047    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"
Oct 30 19:58:03.664312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.664293    4420 scope.go:122] "RemoveContainer" containerID="08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7"
Oct 30 19:58:03.664610 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:03.664587    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7\": container with ID starting with 08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7 not found: ID does not exist" containerID="08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7"
Oct 30 19:58:03.664658 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.664619    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7"} err="failed to get container status \"08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7\": rpc error: code = NotFound desc = could not find container \"08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7\": container with ID starting with 08179b82ba661aed73bb6c0814e75a26c522455f66e01c1989be83de41770eb7 not found: ID does not exist"
Oct 30 19:58:03.825840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.821900    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"]
Oct 30 19:58:03.825840 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:03.823687    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-daa1ac-759788fd96-6ksfh"]
Oct 30 19:58:04.896686 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:04.896651    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa6ea6a-ca2f-49e7-9a94-75a001e56a42" path="/var/lib/kubelet/pods/6fa6ea6a-ca2f-49e7-9a94-75a001e56a42/volumes"
Oct 30 19:58:05.196056 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:05.195976    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:58:12.532479 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:12.532450    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"]
Oct 30 19:58:12.532933 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:12.532672    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy" containerID="cri-o://63764c9cf3a7abe8d7234808caf1716a79b160ae58509226a05b06509ed00381" gracePeriod=30
Oct 30 19:58:12.696022 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:12.695987    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"]
Oct 30 19:58:12.727595 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:12.727538    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"]
Oct 30 19:58:15.112079 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:15.112048    4420 patch_prober.go:29] interesting pod/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:58:15.112463 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:15.112109    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:58:17.766787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.766756    4420 generic.go:356] "Generic (PLEG): container finished" podID="415f5443-053d-4e88-814d-91952beb32f5" containerID="63764c9cf3a7abe8d7234808caf1716a79b160ae58509226a05b06509ed00381" exitCode=0
Oct 30 19:58:17.767237 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.766800    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" event={"ID":"415f5443-053d-4e88-814d-91952beb32f5","Type":"ContainerDied","Data":"63764c9cf3a7abe8d7234808caf1716a79b160ae58509226a05b06509ed00381"}
Oct 30 19:58:17.767237 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.766897    4420 scope.go:122] "RemoveContainer" containerID="af82ba18af013c0ce5d8a6004368c71f9556a32f48e2ec64559596ae2b03d4f6"
Oct 30 19:58:17.824894 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.824870    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:58:17.955291 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955267    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-socket\") pod \"415f5443-053d-4e88-814d-91952beb32f5\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") "
Oct 30 19:58:17.955406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955298    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-envoy\") pod \"415f5443-053d-4e88-814d-91952beb32f5\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") "
Oct 30 19:58:17.955406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955373    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-istio-token\" (UniqueName: \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-istio-token\") pod \"415f5443-053d-4e88-814d-91952beb32f5\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") "
Oct 30 19:58:17.955503 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955419    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415f5443-053d-4e88-814d-91952beb32f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415f5443-053d-4e88-814d-91952beb32f5-istiod-ca-cert\") pod \"415f5443-053d-4e88-814d-91952beb32f5\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") "
Oct 30 19:58:17.955503 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955448    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-certs\") pod \"415f5443-053d-4e88-814d-91952beb32f5\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") "
Oct 30 19:58:17.955503 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955476    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-socket" pod "415f5443-053d-4e88-814d-91952beb32f5" (UID: "415f5443-053d-4e88-814d-91952beb32f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:17.955503 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955490    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-kube-api-access-ldqmp\" (UniqueName: \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-kube-api-access-ldqmp\") pod \"415f5443-053d-4e88-814d-91952beb32f5\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") "
Oct 30 19:58:17.955690 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955528    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-credential-socket\") pod \"415f5443-053d-4e88-814d-91952beb32f5\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") "
Oct 30 19:58:17.955690 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955548    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-data\") pod \"415f5443-053d-4e88-814d-91952beb32f5\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") "
Oct 30 19:58:17.955690 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955576    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/415f5443-053d-4e88-814d-91952beb32f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415f5443-053d-4e88-814d-91952beb32f5-istio-podinfo\") pod \"415f5443-053d-4e88-814d-91952beb32f5\" (UID: \"415f5443-053d-4e88-814d-91952beb32f5\") "
Oct 30 19:58:17.955889 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955740    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-certs" pod "415f5443-053d-4e88-814d-91952beb32f5" (UID: "415f5443-053d-4e88-814d-91952beb32f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:17.955889 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955861    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-credential-socket" pod "415f5443-053d-4e88-814d-91952beb32f5" (UID: "415f5443-053d-4e88-814d-91952beb32f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:17.956006 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.955905    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415f5443-053d-4e88-814d-91952beb32f5-istiod-ca-cert" pod "415f5443-053d-4e88-814d-91952beb32f5" (UID: "415f5443-053d-4e88-814d-91952beb32f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:17.956232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.956164    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-data" pod "415f5443-053d-4e88-814d-91952beb32f5" (UID: "415f5443-053d-4e88-814d-91952beb32f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:17.956344 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.956330    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:17.956411 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.956352    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:17.956411 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.956366    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:17.956411 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.956381    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/415f5443-053d-4e88-814d-91952beb32f5-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:17.956411 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.956396    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:17.957608 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.957585    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-envoy" pod "415f5443-053d-4e88-814d-91952beb32f5" (UID: "415f5443-053d-4e88-814d-91952beb32f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:17.957909 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.957885    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/415f5443-053d-4e88-814d-91952beb32f5-istio-podinfo" pod "415f5443-053d-4e88-814d-91952beb32f5" (UID: "415f5443-053d-4e88-814d-91952beb32f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:58:17.958304 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.958279    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-istio-token" pod "415f5443-053d-4e88-814d-91952beb32f5" (UID: "415f5443-053d-4e88-814d-91952beb32f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:17.958304 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:17.958287    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-kube-api-access-ldqmp" pod "415f5443-053d-4e88-814d-91952beb32f5" (UID: "415f5443-053d-4e88-814d-91952beb32f5"). InnerVolumeSpecName "kube-api-access-ldqmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:18.057922 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:18.057725    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:18.057922 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:18.057756    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldqmp\" (UniqueName: \"kubernetes.io/projected/415f5443-053d-4e88-814d-91952beb32f5-kube-api-access-ldqmp\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:18.057922 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:18.057773    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/415f5443-053d-4e88-814d-91952beb32f5-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:18.057922 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:18.057787    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/415f5443-053d-4e88-814d-91952beb32f5-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:18.774637 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:18.774611    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"
Oct 30 19:58:18.775051 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:18.774648    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6" event={"ID":"415f5443-053d-4e88-814d-91952beb32f5","Type":"ContainerDied","Data":"47faa47b4b910856bf5f1f3740428ea22d1a431ffe5fa55fef5b823959499cfd"}
Oct 30 19:58:18.775051 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:18.774701    4420 scope.go:122] "RemoveContainer" containerID="63764c9cf3a7abe8d7234808caf1716a79b160ae58509226a05b06509ed00381"
Oct 30 19:58:19.000215 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:19.000192    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"]
Oct 30 19:58:19.048513 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:19.048447    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-daa1ac-openshift-default-7dc9d6dfb5-pk7n6"]
Oct 30 19:58:20.897011 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:20.896980    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415f5443-053d-4e88-814d-91952beb32f5" path="/var/lib/kubelet/pods/415f5443-053d-4e88-814d-91952beb32f5/volumes"
Oct 30 19:58:24.025506 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.025477    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"]
Oct 30 19:58:24.026302 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.026283    4420 cpu_manager.go:414] "containerMap: removing container" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy"
Oct 30 19:58:24.026351 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.026305    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy"
Oct 30 19:58:24.026351 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.026321    4420 cpu_manager.go:414] "containerMap: removing container" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy"
Oct 30 19:58:24.026351 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.026328    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy"
Oct 30 19:58:24.026351 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.026336    4420 cpu_manager.go:414] "containerMap: removing container" podUID="6fa6ea6a-ca2f-49e7-9a94-75a001e56a42" containerName="maas-api"
Oct 30 19:58:24.026351 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.026341    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa6ea6a-ca2f-49e7-9a94-75a001e56a42" containerName="maas-api"
Oct 30 19:58:24.026504 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.026442    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy"
Oct 30 19:58:24.026504 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.026450    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="415f5443-053d-4e88-814d-91952beb32f5" containerName="istio-proxy"
Oct 30 19:58:24.026504 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.026458    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa6ea6a-ca2f-49e7-9a94-75a001e56a42" containerName="maas-api"
Oct 30 19:58:24.030146 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.030120    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.033702 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.033679    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwzw\" (UniqueName: \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-kube-api-access-6qwzw\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.033809 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.033736    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.033905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.033803    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.033975 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.033959    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.034087 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.034067    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.034195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.034149    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.034195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.034179    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.034281 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.034195    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.034281 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.034245    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.135368 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.135326    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.135551 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.135447    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwzw\" (UniqueName: \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-kube-api-access-6qwzw\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.135872 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.135799    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.135986 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.135807    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.135986 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.135928    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.136115 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.136010    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.136115 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.136063    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.136115 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.136071    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.136259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.136188    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.136259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.136233    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.136259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.136239    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.136438 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.136256    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.136438 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.136295    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.137184 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.137161    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.138179 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.138155    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"]
Oct 30 19:58:24.139149 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.139103    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.139739 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.139694    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.180106 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.180087    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-dd9b8d79-openshift-default-dockercfg-nk77p\""
Oct 30 19:58:24.191671 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.191650    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.192253 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.192227    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwzw\" (UniqueName: \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-kube-api-access-6qwzw\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.343272 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.343209    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:24.781299 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.781273    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"]
Oct 30 19:58:24.782334 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:58:24.782306    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb886fa6_4fe0_47e5_9cd1_a1ec2ee6b062.slice/crio-3798ccb7516ce34012d6c9448c8aeb0c5d9aa23efa8bc4ccd12677209cc2e9dc WatchSource:0}: Error finding container 3798ccb7516ce34012d6c9448c8aeb0c5d9aa23efa8bc4ccd12677209cc2e9dc: Status 404 returned error can't find the container with id 3798ccb7516ce34012d6c9448c8aeb0c5d9aa23efa8bc4ccd12677209cc2e9dc
Oct 30 19:58:24.785057 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.785028    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:24.785184 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.785089    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:24.785184 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.785133    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:24.815461 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:24.815436    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" event={"ID":"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062","Type":"ContainerStarted","Data":"3798ccb7516ce34012d6c9448c8aeb0c5d9aa23efa8bc4ccd12677209cc2e9dc"}
Oct 30 19:58:25.823530 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:25.823499    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" event={"ID":"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062","Type":"ContainerStarted","Data":"317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8"}
Oct 30 19:58:26.344232 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:26.344204    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:26.345911 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:26.345884    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:58:26.346041 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:26.345948    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:58:27.344442 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:27.344410    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:58:27.344927 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:27.344460    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:58:27.728665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:27.728609    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main" containerID="cri-o://7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6" gracePeriod=45
Oct 30 19:58:27.750395 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:27.750367    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main" containerID="cri-o://1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227" gracePeriod=45
Oct 30 19:58:28.115474 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.115453    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:58:28.118515 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.118498    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:58:28.207741 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.207718    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-kube-api-access-8b6s5\" (UniqueName: \"kubernetes.io/projected/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-kube-api-access-8b6s5\") pod \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") "
Oct 30 19:58:28.207899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.207788    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-dshm\") pod \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") "
Oct 30 19:58:28.207899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.207842    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tmp-dir\") pod \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") "
Oct 30 19:58:28.207899 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.207870    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7fa2065-d5ee-4487-b82e-021ade5b8378-kube-api-access-7d7rv\" (UniqueName: \"kubernetes.io/projected/d7fa2065-d5ee-4487-b82e-021ade5b8378-kube-api-access-7d7rv\") pod \"d7fa2065-d5ee-4487-b82e-021ade5b8378\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") "
Oct 30 19:58:28.208075 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.207913    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-model-cache\") pod \"d7fa2065-d5ee-4487-b82e-021ade5b8378\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") "
Oct 30 19:58:28.208075 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.207937    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-tmp-dir\") pod \"d7fa2065-d5ee-4487-b82e-021ade5b8378\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") "
Oct 30 19:58:28.208075 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.207980    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-home\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-home\") pod \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") "
Oct 30 19:58:28.208075 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208049    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-model-cache\") pod \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") "
Oct 30 19:58:28.208075 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208054    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tmp-dir" pod "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" (UID: "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.208289 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208118    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-home\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-home\") pod \"d7fa2065-d5ee-4487-b82e-021ade5b8378\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") "
Oct 30 19:58:28.208289 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208153    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7fa2065-d5ee-4487-b82e-021ade5b8378-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7fa2065-d5ee-4487-b82e-021ade5b8378-tls-certs\") pod \"d7fa2065-d5ee-4487-b82e-021ade5b8378\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") "
Oct 30 19:58:28.208289 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208198    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-dshm\") pod \"d7fa2065-d5ee-4487-b82e-021ade5b8378\" (UID: \"d7fa2065-d5ee-4487-b82e-021ade5b8378\") "
Oct 30 19:58:28.208289 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208199    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-model-cache" pod "d7fa2065-d5ee-4487-b82e-021ade5b8378" (UID: "d7fa2065-d5ee-4487-b82e-021ade5b8378"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.208289 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208204    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-home" pod "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" (UID: "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.208289 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208213    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-tmp-dir" pod "d7fa2065-d5ee-4487-b82e-021ade5b8378" (UID: "d7fa2065-d5ee-4487-b82e-021ade5b8378"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.208289 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208223    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tls-certs\") pod \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\" (UID: \"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299\") "
Oct 30 19:58:28.208614 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208356    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-home" pod "d7fa2065-d5ee-4487-b82e-021ade5b8378" (UID: "d7fa2065-d5ee-4487-b82e-021ade5b8378"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.208614 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208396    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-model-cache" pod "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" (UID: "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.208876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208805    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.208876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208851    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.208876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208865    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.209238 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208889    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.209238 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208916    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.209238 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.208997    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.210619 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.210597    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-dshm" pod "d7fa2065-d5ee-4487-b82e-021ade5b8378" (UID: "d7fa2065-d5ee-4487-b82e-021ade5b8378"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.210746 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.210714    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7fa2065-d5ee-4487-b82e-021ade5b8378-tls-certs" pod "d7fa2065-d5ee-4487-b82e-021ade5b8378" (UID: "d7fa2065-d5ee-4487-b82e-021ade5b8378"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:28.210857 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.210808    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-dshm" pod "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" (UID: "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:28.210916 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.210866    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7fa2065-d5ee-4487-b82e-021ade5b8378-kube-api-access-7d7rv" pod "d7fa2065-d5ee-4487-b82e-021ade5b8378" (UID: "d7fa2065-d5ee-4487-b82e-021ade5b8378"). InnerVolumeSpecName "kube-api-access-7d7rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:28.211147 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.211123    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-kube-api-access-8b6s5" pod "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" (UID: "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299"). InnerVolumeSpecName "kube-api-access-8b6s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:28.211186 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.211163    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tls-certs" pod "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" (UID: "3c631f6b-ff1f-40c2-a8b3-d3b605bc6299"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:28.231968 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.231928    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" podStartSLOduration=5.231912344 podStartE2EDuration="5.231912344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:26.037931426 +0000 UTC m=+2553.249148570" watchObservedRunningTime="2026-10-30 19:58:28.231912344 +0000 UTC m=+2555.443129490"
Oct 30 19:58:28.309980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.309889    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b6s5\" (UniqueName: \"kubernetes.io/projected/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-kube-api-access-8b6s5\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.309980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.309933    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.309980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.309943    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7rv\" (UniqueName: \"kubernetes.io/projected/d7fa2065-d5ee-4487-b82e-021ade5b8378-kube-api-access-7d7rv\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.309980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.309952    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7fa2065-d5ee-4487-b82e-021ade5b8378-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.309980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.309960    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7fa2065-d5ee-4487-b82e-021ade5b8378-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.309980 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.309969    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:28.344604 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.344564    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:58:28.345206 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.344631    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:58:28.845929 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.845887    4420 generic.go:356] "Generic (PLEG): container finished" podID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerID="7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6" exitCode=0
Oct 30 19:58:28.846092 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.846046    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" event={"ID":"d7fa2065-d5ee-4487-b82e-021ade5b8378","Type":"ContainerDied","Data":"7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6"}
Oct 30 19:58:28.846092 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.846075    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz" event={"ID":"d7fa2065-d5ee-4487-b82e-021ade5b8378","Type":"ContainerDied","Data":"1be8a2c95d98a9af393d9feeed7378288b8aae532e13a11325745084dc101260"}
Oct 30 19:58:28.846092 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.846090    4420 scope.go:122] "RemoveContainer" containerID="7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6"
Oct 30 19:58:28.846228 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.846198    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"
Oct 30 19:58:28.849956 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.849930    4420 generic.go:356] "Generic (PLEG): container finished" podID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerID="1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227" exitCode=0
Oct 30 19:58:28.850027 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.850010    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" event={"ID":"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299","Type":"ContainerDied","Data":"1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227"}
Oct 30 19:58:28.850076 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.850041    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs" event={"ID":"3c631f6b-ff1f-40c2-a8b3-d3b605bc6299","Type":"ContainerDied","Data":"b08292855379e289b3f03aaa1464e3f9a8d7d670d2ff54addea402e18564dc45"}
Oct 30 19:58:28.850122 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.850108    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"
Oct 30 19:58:28.868030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.865805    4420 scope.go:122] "RemoveContainer" containerID="7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6"
Oct 30 19:58:28.868030 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:28.866161    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6\": container with ID starting with 7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6 not found: ID does not exist" containerID="7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6"
Oct 30 19:58:28.868030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.866195    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6"} err="failed to get container status \"7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6\": rpc error: code = NotFound desc = could not find container \"7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6\": container with ID starting with 7534908ba179d1e2784d2d258d518a92fbe4a5af76625ba667b249b75cbc81c6 not found: ID does not exist"
Oct 30 19:58:28.868030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.866220    4420 scope.go:122] "RemoveContainer" containerID="1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227"
Oct 30 19:58:28.895720 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.895700    4420 scope.go:122] "RemoveContainer" containerID="1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227"
Oct 30 19:58:28.896173 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:28.896147    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227\": container with ID starting with 1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227 not found: ID does not exist" containerID="1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227"
Oct 30 19:58:28.896239 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:28.896185    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227"} err="failed to get container status \"1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227\": rpc error: code = NotFound desc = could not find container \"1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227\": container with ID starting with 1a11a9f0d54dd905888b1c4f33b2874c551e618520d5141e15159357b0469227 not found: ID does not exist"
Oct 30 19:58:29.012569 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:29.012491    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"]
Oct 30 19:58:29.016773 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:29.016750    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/premium-w1-d52df0-kserve-7cd9655bdd-lmjvs"]
Oct 30 19:58:29.140062 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:29.140036    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"]
Oct 30 19:58:29.143014 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:29.142991    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d52df0/distinct-w1-d52df0-kserve-756846d4bc-vvthz"]
Oct 30 19:58:29.344532 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:29.344453    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:58:29.344780 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:29.344755    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:58:30.344498 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:30.344461    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:58:30.344675 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:30.344531    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:58:30.897708 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:30.897671    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" path="/var/lib/kubelet/pods/3c631f6b-ff1f-40c2-a8b3-d3b605bc6299/volumes"
Oct 30 19:58:30.898486 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:30.898324    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" path="/var/lib/kubelet/pods/d7fa2065-d5ee-4487-b82e-021ade5b8378/volumes"
Oct 30 19:58:31.344131 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:31.344054    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 19:58:31.344131 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:31.344121    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 19:58:32.348148 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:32.348122    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:32.348596 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:32.348376    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:32.349133 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:32.349117    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 19:58:34.541727 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.541657    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"]
Oct 30 19:58:34.542104 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.541890    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm" podUID="6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c" containerName="maas-api" containerID="cri-o://896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15" gracePeriod=30
Oct 30 19:58:34.844674 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.844647    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:58:34.916199 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.916167    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-metrics-certs\") pod \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") "
Oct 30 19:58:34.916356 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.916254    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-maas-api-tls\") pod \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") "
Oct 30 19:58:34.916423 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.916360    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-kube-api-access-x4kjj\" (UniqueName: \"kubernetes.io/projected/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-kube-api-access-x4kjj\") pod \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") "
Oct 30 19:58:34.916423 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.916394    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-openshift-service-ca\") pod \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\" (UID: \"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c\") "
Oct 30 19:58:34.917085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.916737    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-openshift-service-ca" pod "6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c" (UID: "6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:34.917085 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.917008    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:34.917974 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.917947    4420 generic.go:356] "Generic (PLEG): container finished" podID="6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c" containerID="896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15" exitCode=0
Oct 30 19:58:34.918071 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.917995    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"
Oct 30 19:58:34.918071 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.918010    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm" event={"ID":"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c","Type":"ContainerDied","Data":"896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15"}
Oct 30 19:58:34.918071 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.918055    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm" event={"ID":"6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c","Type":"ContainerDied","Data":"aeb1a6b14e60b61adbc433b5a6584e317e21d280ecae0ee2cefa4e4587b4acdc"}
Oct 30 19:58:34.918241 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.918082    4420 scope.go:122] "RemoveContainer" containerID="896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15"
Oct 30 19:58:34.918721 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.918698    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-maas-api-tls" pod "6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c" (UID: "6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:34.918998 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.918976    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-metrics-certs" pod "6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c" (UID: "6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:34.919065 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.919013    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-kube-api-access-x4kjj" pod "6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c" (UID: "6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c"). InnerVolumeSpecName "kube-api-access-x4kjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:34.945295 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.945274    4420 scope.go:122] "RemoveContainer" containerID="896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15"
Oct 30 19:58:34.945626 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:34.945600    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15\": container with ID starting with 896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15 not found: ID does not exist" containerID="896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15"
Oct 30 19:58:34.945716 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:34.945638    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15"} err="failed to get container status \"896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15\": rpc error: code = NotFound desc = could not find container \"896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15\": container with ID starting with 896e382c2cfa3185140b4e67b8a6832450cc865d4c8c66802531ecd722c79d15 not found: ID does not exist"
Oct 30 19:58:35.018401 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:35.018369    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4kjj\" (UniqueName: \"kubernetes.io/projected/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-kube-api-access-x4kjj\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:35.018557 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:35.018497    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:35.018557 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:35.018515    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:35.237962 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:35.237918    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:58:35.352158 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:35.352128    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"]
Oct 30 19:58:35.357063 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:35.357038    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6873e0-57f79bf6dd-5dzlm"]
Oct 30 19:58:36.896724 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:36.896691    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c" path="/var/lib/kubelet/pods/6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c/volumes"
Oct 30 19:58:39.122459 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.122420    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"]
Oct 30 19:58:39.123324 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.123301    4420 cpu_manager.go:414] "containerMap: removing container" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main"
Oct 30 19:58:39.123324 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.123325    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main"
Oct 30 19:58:39.123490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.123349    4420 cpu_manager.go:414] "containerMap: removing container" podUID="6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c" containerName="maas-api"
Oct 30 19:58:39.123490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.123356    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c" containerName="maas-api"
Oct 30 19:58:39.123490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.123385    4420 cpu_manager.go:414] "containerMap: removing container" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main"
Oct 30 19:58:39.123490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.123392    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main"
Oct 30 19:58:39.123697 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.123544    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7fa2065-d5ee-4487-b82e-021ade5b8378" containerName="main"
Oct 30 19:58:39.123697 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.123568    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c631f6b-ff1f-40c2-a8b3-d3b605bc6299" containerName="main"
Oct 30 19:58:39.123697 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.123581    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd1e8d2-cff8-4a83-a24d-7e5e19d8ad9c" containerName="maas-api"
Oct 30 19:58:39.128376 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.128355    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.227750 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.227713    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"]
Oct 30 19:58:39.242603 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.242575    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"]
Oct 30 19:58:39.242858 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.242806    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy" containerID="cri-o://86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f" gracePeriod=30
Oct 30 19:58:39.282277 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.282247    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2thqk\" (UniqueName: \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-kube-api-access-2thqk\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.282488 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.282465    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.282722 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.282621    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.282722 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.282678    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.282972 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.282905    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.282972 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.282962    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.283094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.282985    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.283094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.283019    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.283094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.283053    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384101    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384165    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384185 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384186    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384476 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384217    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384476 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384262    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384476 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384342    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thqk\" (UniqueName: \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-kube-api-access-2thqk\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384476 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384363    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384476 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384389    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384476 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384444    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384701    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-data\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384741    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.384928 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384866    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-certs\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.385136 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.384990    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-credential-socket\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.385468 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.385440    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istiod-ca-cert\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.386772 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.386748    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-envoy\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.386909 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.386854    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-podinfo\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.441317 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.441295    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-token\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.441443 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.441362    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2thqk\" (UniqueName: \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-kube-api-access-2thqk\") pod \"e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") " pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.442225 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.442202    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:39.870389 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.870365    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"]
Oct 30 19:58:39.871429 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:58:39.871404    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12b03fa_7887_40d9_bcc6_8c71f815ad0b.slice/crio-a54e215dffc080ca65ab83ff2930efbbf3fc989767ea625ec900009f4d2d08d4 WatchSource:0}: Error finding container a54e215dffc080ca65ab83ff2930efbbf3fc989767ea625ec900009f4d2d08d4: Status 404 returned error can't find the container with id a54e215dffc080ca65ab83ff2930efbbf3fc989767ea625ec900009f4d2d08d4
Oct 30 19:58:39.873805 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.873776    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:39.873876 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.873860    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:39.873913 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.873888    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 19:58:39.953574 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.953544    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" event={"ID":"b12b03fa-7887-40d9-bcc6-8c71f815ad0b","Type":"ContainerStarted","Data":"e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f"}
Oct 30 19:58:39.953714 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:39.953602    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" event={"ID":"b12b03fa-7887-40d9-bcc6-8c71f815ad0b","Type":"ContainerStarted","Data":"a54e215dffc080ca65ab83ff2930efbbf3fc989767ea625ec900009f4d2d08d4"}
Oct 30 19:58:40.075887 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:40.075833    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" podStartSLOduration=1.075798326 podStartE2EDuration="1.075798326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:40.073332086 +0000 UTC m=+2567.284549267" watchObservedRunningTime="2026-10-30 19:58:40.075798326 +0000 UTC m=+2567.287015472"
Oct 30 19:58:40.443134 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:40.443105    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:40.444303 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:40.444283    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:58:40.444354 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:40.444329    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 19:58:41.443330 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:41.443301    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:58:41.443700 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:41.443350    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 19:58:42.443432 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:42.443400    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:58:42.443801 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:42.443448    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 19:58:43.442777 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:43.442745    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:58:43.442962 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:43.442793    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 19:58:44.444386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.444073    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:58:44.444386 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.444117    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 19:58:44.555055 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.555028    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:58:44.685869 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.684566    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-certs\") pod \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") "
Oct 30 19:58:44.685869 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.684639    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-credential-socket\") pod \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") "
Oct 30 19:58:44.685869 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.684710    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-token\" (UniqueName: \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-token\") pod \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") "
Oct 30 19:58:44.685869 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.684803    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-envoy\") pod \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") "
Oct 30 19:58:44.685869 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.684886    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-podinfo\") pod \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") "
Oct 30 19:58:44.685869 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.684951    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-kube-api-access-6ml6g\" (UniqueName: \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-kube-api-access-6ml6g\") pod \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") "
Oct 30 19:58:44.685869 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.685006    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-socket\") pod \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") "
Oct 30 19:58:44.685869 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.685051    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-data\") pod \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") "
Oct 30 19:58:44.685869 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.685072    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istiod-ca-cert\") pod \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\" (UID: \"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6\") "
Oct 30 19:58:44.686481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.686003    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istiod-ca-cert" pod "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" (UID: "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:44.686481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.686305    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-socket" pod "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" (UID: "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:44.686481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.686355    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-data" pod "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" (UID: "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:44.686481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.686479    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-credential-socket" pod "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" (UID: "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:44.686719 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.686536    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-certs" pod "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" (UID: "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:44.688975 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.688947    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-envoy" pod "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" (UID: "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:44.689428 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.689387    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-token" pod "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" (UID: "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:44.689617 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.689583    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-podinfo" pod "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" (UID: "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:58:44.689892 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.689868    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-kube-api-access-6ml6g" pod "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" (UID: "73dcc5e9-23c0-48c9-9fe3-8b30ea248db6"). InnerVolumeSpecName "kube-api-access-6ml6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:44.786958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.786886    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:44.786958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.786910    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:44.786958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.786918    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:44.786958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.786926    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:44.786958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.786935    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:44.786958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.786943    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:44.786958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.786951    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:44.786958 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.786960    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:44.787315 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:44.786973    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ml6g\" (UniqueName: \"kubernetes.io/projected/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6-kube-api-access-6ml6g\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:45.005716 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:45.005679    4420 generic.go:356] "Generic (PLEG): container finished" podID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerID="86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f" exitCode=0
Oct 30 19:58:45.005924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:45.005736    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"
Oct 30 19:58:45.005924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:45.005758    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" event={"ID":"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6","Type":"ContainerDied","Data":"86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f"}
Oct 30 19:58:45.005924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:45.005830    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4" event={"ID":"73dcc5e9-23c0-48c9-9fe3-8b30ea248db6","Type":"ContainerDied","Data":"a7e86214210b77384725cfc9dd04edd828eb2fcf29d296c7cae777fe2c675c88"}
Oct 30 19:58:45.005924 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:45.005860    4420 scope.go:122] "RemoveContainer" containerID="86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f"
Oct 30 19:58:45.016499 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:45.016480    4420 scope.go:122] "RemoveContainer" containerID="86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f"
Oct 30 19:58:45.016768 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:45.016749    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f\": container with ID starting with 86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f not found: ID does not exist" containerID="86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f"
Oct 30 19:58:45.016846 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:45.016780    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f"} err="failed to get container status \"86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f\": rpc error: code = NotFound desc = could not find container \"86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f\": container with ID starting with 86d863b62a582f5ca3f5303d5e6a4f3fd8a79dbae98aeba8ee85fc16c805999f not found: ID does not exist"
Oct 30 19:58:45.195369 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:45.195278    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"]
Oct 30 19:58:45.241973 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:45.241948    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-vzbz4"]
Oct 30 19:58:45.443090 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:45.443061    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 19:58:45.443259 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:45.443130    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 19:58:46.457637 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:46.457584    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:46.458995 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:46.458967    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:46.460538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:46.460515    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 19:58:46.464475 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:46.464453    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"]
Oct 30 19:58:46.464701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:46.464669    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy" containerID="cri-o://3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8" gracePeriod=30
Oct 30 19:58:46.896936 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:46.896909    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" path="/var/lib/kubelet/pods/73dcc5e9-23c0-48c9-9fe3-8b30ea248db6/volumes"
Oct 30 19:58:51.732909 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.732851    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:58:51.888275 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888253    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-certs\") pod \"72e17a94-a1f3-4273-a8f5-458b8d478a16\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") "
Oct 30 19:58:51.888433 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888297    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-kube-api-access-k5bx8\" (UniqueName: \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-kube-api-access-k5bx8\") pod \"72e17a94-a1f3-4273-a8f5-458b8d478a16\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") "
Oct 30 19:58:51.888433 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888321    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72e17a94-a1f3-4273-a8f5-458b8d478a16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e17a94-a1f3-4273-a8f5-458b8d478a16-istiod-ca-cert\") pod \"72e17a94-a1f3-4273-a8f5-458b8d478a16\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") "
Oct 30 19:58:51.888433 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888347    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-socket\") pod \"72e17a94-a1f3-4273-a8f5-458b8d478a16\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") "
Oct 30 19:58:51.888577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888444    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-certs" pod "72e17a94-a1f3-4273-a8f5-458b8d478a16" (UID: "72e17a94-a1f3-4273-a8f5-458b8d478a16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:51.888577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888498    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-token\" (UniqueName: \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-token\") pod \"72e17a94-a1f3-4273-a8f5-458b8d478a16\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") "
Oct 30 19:58:51.888577 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888570    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-envoy\") pod \"72e17a94-a1f3-4273-a8f5-458b8d478a16\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") "
Oct 30 19:58:51.888702 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888624    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-socket" pod "72e17a94-a1f3-4273-a8f5-458b8d478a16" (UID: "72e17a94-a1f3-4273-a8f5-458b8d478a16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:51.888702 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888648    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-credential-socket\") pod \"72e17a94-a1f3-4273-a8f5-458b8d478a16\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") "
Oct 30 19:58:51.888801 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888702    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-podinfo\") pod \"72e17a94-a1f3-4273-a8f5-458b8d478a16\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") "
Oct 30 19:58:51.888897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888808    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72e17a94-a1f3-4273-a8f5-458b8d478a16-istiod-ca-cert" pod "72e17a94-a1f3-4273-a8f5-458b8d478a16" (UID: "72e17a94-a1f3-4273-a8f5-458b8d478a16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:51.888897 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.888854    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-data\") pod \"72e17a94-a1f3-4273-a8f5-458b8d478a16\" (UID: \"72e17a94-a1f3-4273-a8f5-458b8d478a16\") "
Oct 30 19:58:51.889054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.889017    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-credential-socket" pod "72e17a94-a1f3-4273-a8f5-458b8d478a16" (UID: "72e17a94-a1f3-4273-a8f5-458b8d478a16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:51.889054 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.889032    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-data" pod "72e17a94-a1f3-4273-a8f5-458b8d478a16" (UID: "72e17a94-a1f3-4273-a8f5-458b8d478a16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:51.889502 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.889470    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:51.889631 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.889612    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:51.889701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.889637    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:51.889701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.889651    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72e17a94-a1f3-4273-a8f5-458b8d478a16-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:51.889701 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.889665    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:51.890671 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.890645    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-kube-api-access-k5bx8" pod "72e17a94-a1f3-4273-a8f5-458b8d478a16" (UID: "72e17a94-a1f3-4273-a8f5-458b8d478a16"). InnerVolumeSpecName "kube-api-access-k5bx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:51.891094 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.891075    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-podinfo" pod "72e17a94-a1f3-4273-a8f5-458b8d478a16" (UID: "72e17a94-a1f3-4273-a8f5-458b8d478a16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:58:51.891276 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.891263    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-envoy" pod "72e17a94-a1f3-4273-a8f5-458b8d478a16" (UID: "72e17a94-a1f3-4273-a8f5-458b8d478a16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:58:51.891330 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.891292    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-token" pod "72e17a94-a1f3-4273-a8f5-458b8d478a16" (UID: "72e17a94-a1f3-4273-a8f5-458b8d478a16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:51.991225 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.991159    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5bx8\" (UniqueName: \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-kube-api-access-k5bx8\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:51.991225 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.991185    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:51.991225 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.991199    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:51.991225 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:51.991212    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72e17a94-a1f3-4273-a8f5-458b8d478a16-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:52.056581 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.056552    4420 generic.go:356] "Generic (PLEG): container finished" podID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerID="3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8" exitCode=0
Oct 30 19:58:52.056707 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.056602    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"
Oct 30 19:58:52.056707 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.056615    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" event={"ID":"72e17a94-a1f3-4273-a8f5-458b8d478a16","Type":"ContainerDied","Data":"3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8"}
Oct 30 19:58:52.056707 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.056655    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26" event={"ID":"72e17a94-a1f3-4273-a8f5-458b8d478a16","Type":"ContainerDied","Data":"4b7c4de992045661ac065d323d6b01a7ec5d2fc960359bbe7266d16b2838bc08"}
Oct 30 19:58:52.056707 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.056674    4420 scope.go:122] "RemoveContainer" containerID="3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8"
Oct 30 19:58:52.069114 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.069096    4420 scope.go:122] "RemoveContainer" containerID="714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553"
Oct 30 19:58:52.079051 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.079018    4420 scope.go:122] "RemoveContainer" containerID="3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8"
Oct 30 19:58:52.079341 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:52.079320    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8\": container with ID starting with 3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8 not found: ID does not exist" containerID="3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8"
Oct 30 19:58:52.079434 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.079368    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8"} err="failed to get container status \"3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8\": rpc error: code = NotFound desc = could not find container \"3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8\": container with ID starting with 3cfcbae88f2e8091f790ed9a62821f70a74bc79858d43243db99fc7d69192dc8 not found: ID does not exist"
Oct 30 19:58:52.079434 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.079386    4420 scope.go:122] "RemoveContainer" containerID="714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553"
Oct 30 19:58:52.079644 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:52.079625    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553\": container with ID starting with 714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553 not found: ID does not exist" containerID="714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553"
Oct 30 19:58:52.079684 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.079650    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553"} err="failed to get container status \"714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553\": rpc error: code = NotFound desc = could not find container \"714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553\": container with ID starting with 714a52895415b6d705a3f50766b04d7a5d28c4b6a194b5be2fa65d74e5162553 not found: ID does not exist"
Oct 30 19:58:52.221711 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.221685    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"]
Oct 30 19:58:52.226305 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.226280    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6873e0-openshift-default-765884b6df-bxb26"]
Oct 30 19:58:52.895966 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:52.895940    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" path="/var/lib/kubelet/pods/72e17a94-a1f3-4273-a8f5-458b8d478a16/volumes"
Oct 30 19:58:53.017187 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.017157    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"]
Oct 30 19:58:53.017741 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.017728    4420 cpu_manager.go:414] "containerMap: removing container" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy"
Oct 30 19:58:53.017801 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.017742    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy"
Oct 30 19:58:53.017801 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.017754    4420 cpu_manager.go:414] "containerMap: removing container" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy"
Oct 30 19:58:53.017801 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.017758    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy"
Oct 30 19:58:53.017801 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.017773    4420 cpu_manager.go:414] "containerMap: removing container" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy"
Oct 30 19:58:53.017801 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.017778    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy"
Oct 30 19:58:53.018072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.017909    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy"
Oct 30 19:58:53.018072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.017920    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="72e17a94-a1f3-4273-a8f5-458b8d478a16" containerName="istio-proxy"
Oct 30 19:58:53.018072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.017927    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="73dcc5e9-23c0-48c9-9fe3-8b30ea248db6" containerName="istio-proxy"
Oct 30 19:58:53.022637 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.022616    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.075476 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.075184    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-132119-serving-cert\""
Oct 30 19:58:53.078688 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.078664    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"]
Oct 30 19:58:53.102610 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.102587    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.102714 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.102638    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-metrics-certs\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.102895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.102875    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-maas-api-tls\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.102984 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.102931    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pg7f\" (UniqueName: \"kubernetes.io/projected/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-kube-api-access-5pg7f\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.204063 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.203995    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-metrics-certs\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.204202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.204140    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-maas-api-tls\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.204202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.204168    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pg7f\" (UniqueName: \"kubernetes.io/projected/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-kube-api-access-5pg7f\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.204274 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.204259    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.204305 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:53.204277    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-serving-cert: secret "maas-api-e2e-inf-a-132119-serving-cert" not found
Oct 30 19:58:53.204363 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:53.204348    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-maas-api-tls podName:0a8fddd9-8d89-4434-9bc1-7dd44e8f959d nodeName:}" failed. No retries permitted until 2026-10-30 19:58:53.704328547 +0000 UTC m=+2580.915545679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-maas-api-tls") pod "maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq" (UID: "0a8fddd9-8d89-4434-9bc1-7dd44e8f959d") : secret "maas-api-e2e-inf-a-132119-serving-cert" not found
Oct 30 19:58:53.204932 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.204914    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.206582 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.206557    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-metrics-certs\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.279324 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.279295    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pg7f\" (UniqueName: \"kubernetes.io/projected/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-kube-api-access-5pg7f\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.710504 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.710477    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-maas-api-tls\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.712782 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.712756    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-maas-api-tls\") pod \"maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:53.937162 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:53.937126    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:54.358947 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:54.358925    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"]
Oct 30 19:58:54.360897 ip-10-0-133-7 kubenswrapper[4420]: W1030 19:58:54.360867    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8fddd9_8d89_4434_9bc1_7dd44e8f959d.slice/crio-498c4f0ca373830bf6e6836a7fa46497602b1a83d4702eac6373a6ea86885bb1 WatchSource:0}: Error finding container 498c4f0ca373830bf6e6836a7fa46497602b1a83d4702eac6373a6ea86885bb1: Status 404 returned error can't find the container with id 498c4f0ca373830bf6e6836a7fa46497602b1a83d4702eac6373a6ea86885bb1
Oct 30 19:58:55.080540 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:55.080497    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq" event={"ID":"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d","Type":"ContainerStarted","Data":"498c4f0ca373830bf6e6836a7fa46497602b1a83d4702eac6373a6ea86885bb1"}
Oct 30 19:58:56.089528 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:56.089500    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq" event={"ID":"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d","Type":"ContainerStarted","Data":"69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e"}
Oct 30 19:58:56.090011 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:56.089561    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:58:56.206571 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:56.206522    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq" podStartSLOduration=3.287800069 podStartE2EDuration="4.20650881s" totalImagesPullingTime="918.708741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:56.203626853 +0000 UTC m=+2583.414843997" watchObservedRunningTime="2026-10-30 19:58:56.20650881 +0000 UTC m=+2583.417725958"
Oct 30 19:58:57.663960 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:57.663929    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"]
Oct 30 19:58:57.664342 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:57.664162    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2" podUID="76483a35-32ab-46c0-9223-4ecfe57bff49" containerName="maas-api" containerID="cri-o://c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73" gracePeriod=30
Oct 30 19:58:57.939723 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:57.939698    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:58:58.069466 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.069431    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76483a35-32ab-46c0-9223-4ecfe57bff49-kube-api-access-8cj5c\" (UniqueName: \"kubernetes.io/projected/76483a35-32ab-46c0-9223-4ecfe57bff49-kube-api-access-8cj5c\") pod \"76483a35-32ab-46c0-9223-4ecfe57bff49\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") "
Oct 30 19:58:58.069669 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.069501    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-metrics-certs\" (UniqueName: \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-metrics-certs\") pod \"76483a35-32ab-46c0-9223-4ecfe57bff49\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") "
Oct 30 19:58:58.069669 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.069569    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76483a35-32ab-46c0-9223-4ecfe57bff49-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76483a35-32ab-46c0-9223-4ecfe57bff49-openshift-service-ca\") pod \"76483a35-32ab-46c0-9223-4ecfe57bff49\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") "
Oct 30 19:58:58.069669 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.069635    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-maas-api-tls\") pod \"76483a35-32ab-46c0-9223-4ecfe57bff49\" (UID: \"76483a35-32ab-46c0-9223-4ecfe57bff49\") "
Oct 30 19:58:58.079734 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.079701    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76483a35-32ab-46c0-9223-4ecfe57bff49-kube-api-access-8cj5c" pod "76483a35-32ab-46c0-9223-4ecfe57bff49" (UID: "76483a35-32ab-46c0-9223-4ecfe57bff49"). InnerVolumeSpecName "kube-api-access-8cj5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:58:58.080542 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.080513    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76483a35-32ab-46c0-9223-4ecfe57bff49-openshift-service-ca" pod "76483a35-32ab-46c0-9223-4ecfe57bff49" (UID: "76483a35-32ab-46c0-9223-4ecfe57bff49"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:58:58.083223 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.083192    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-metrics-certs" pod "76483a35-32ab-46c0-9223-4ecfe57bff49" (UID: "76483a35-32ab-46c0-9223-4ecfe57bff49"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:58.102108 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.102074    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-maas-api-tls" pod "76483a35-32ab-46c0-9223-4ecfe57bff49" (UID: "76483a35-32ab-46c0-9223-4ecfe57bff49"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:58:58.103993 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.103968    4420 generic.go:356] "Generic (PLEG): container finished" podID="76483a35-32ab-46c0-9223-4ecfe57bff49" containerID="c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73" exitCode=0
Oct 30 19:58:58.104118 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.104043    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"
Oct 30 19:58:58.104181 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.104152    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2" event={"ID":"76483a35-32ab-46c0-9223-4ecfe57bff49","Type":"ContainerDied","Data":"c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73"}
Oct 30 19:58:58.104231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.104188    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2" event={"ID":"76483a35-32ab-46c0-9223-4ecfe57bff49","Type":"ContainerDied","Data":"39a232fa212414891f29cfa8659be961aba45cbb24d8e03174ba34d891876696"}
Oct 30 19:58:58.104231 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.104208    4420 scope.go:122] "RemoveContainer" containerID="c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73"
Oct 30 19:58:58.115595 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.115577    4420 scope.go:122] "RemoveContainer" containerID="c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73"
Oct 30 19:58:58.115863 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:58:58.115844    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73\": container with ID starting with c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73 not found: ID does not exist" containerID="c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73"
Oct 30 19:58:58.115919 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.115870    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73"} err="failed to get container status \"c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73\": rpc error: code = NotFound desc = could not find container \"c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73\": container with ID starting with c673264a61b13cf885ff120332e6b52964e37c2fed45c1c92c56d5d4b174fc73 not found: ID does not exist"
Oct 30 19:58:58.171344 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.171281    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cj5c\" (UniqueName: \"kubernetes.io/projected/76483a35-32ab-46c0-9223-4ecfe57bff49-kube-api-access-8cj5c\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.171344 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.171313    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.171344 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.171329    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76483a35-32ab-46c0-9223-4ecfe57bff49-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.171344 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.171341    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76483a35-32ab-46c0-9223-4ecfe57bff49-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:58:58.271317 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.271287    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"]
Oct 30 19:58:58.276696 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.276670    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-86e4a7-6fd76d7979-wkqf2"]
Oct 30 19:58:58.896585 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:58:58.896554    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76483a35-32ab-46c0-9223-4ecfe57bff49" path="/var/lib/kubelet/pods/76483a35-32ab-46c0-9223-4ecfe57bff49/volumes"
Oct 30 19:59:02.100941 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:02.100914    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 19:59:05.196958 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:05.196927    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:59:07.157278 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:07.157241    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"]
Oct 30 19:59:07.157710 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:07.157465    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy" containerID="cri-o://81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7" gracePeriod=30
Oct 30 19:59:09.124919 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.124888    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"]
Oct 30 19:59:09.125352 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.125078    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w" podUID="089b836d-7fcf-4210-9259-856c8d8fde41" containerName="maas-api" containerID="cri-o://2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560" gracePeriod=30
Oct 30 19:59:09.422633 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.422611    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:59:09.439675 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.439645    4420 patch_prober.go:29] interesting pod/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:59:09.439788 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.439701    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:59:09.513441 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.513400    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-metrics-certs\") pod \"089b836d-7fcf-4210-9259-856c8d8fde41\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") "
Oct 30 19:59:09.514186 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.514161    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-maas-api-tls\") pod \"089b836d-7fcf-4210-9259-856c8d8fde41\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") "
Oct 30 19:59:09.515180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.514438    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089b836d-7fcf-4210-9259-856c8d8fde41-kube-api-access-hxrhk\" (UniqueName: \"kubernetes.io/projected/089b836d-7fcf-4210-9259-856c8d8fde41-kube-api-access-hxrhk\") pod \"089b836d-7fcf-4210-9259-856c8d8fde41\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") "
Oct 30 19:59:09.515180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.514505    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089b836d-7fcf-4210-9259-856c8d8fde41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/089b836d-7fcf-4210-9259-856c8d8fde41-openshift-service-ca\") pod \"089b836d-7fcf-4210-9259-856c8d8fde41\" (UID: \"089b836d-7fcf-4210-9259-856c8d8fde41\") "
Oct 30 19:59:09.515180 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.515154    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089b836d-7fcf-4210-9259-856c8d8fde41-openshift-service-ca" pod "089b836d-7fcf-4210-9259-856c8d8fde41" (UID: "089b836d-7fcf-4210-9259-856c8d8fde41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:09.515412 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.515379    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/089b836d-7fcf-4210-9259-856c8d8fde41-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:09.539531 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.525730    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-metrics-certs" pod "089b836d-7fcf-4210-9259-856c8d8fde41" (UID: "089b836d-7fcf-4210-9259-856c8d8fde41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:09.546261 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.546216    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089b836d-7fcf-4210-9259-856c8d8fde41-kube-api-access-hxrhk" pod "089b836d-7fcf-4210-9259-856c8d8fde41" (UID: "089b836d-7fcf-4210-9259-856c8d8fde41"). InnerVolumeSpecName "kube-api-access-hxrhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:09.547073 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.546495    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-maas-api-tls" pod "089b836d-7fcf-4210-9259-856c8d8fde41" (UID: "089b836d-7fcf-4210-9259-856c8d8fde41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:09.616762 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.616733    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:09.616762 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.616760    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/089b836d-7fcf-4210-9259-856c8d8fde41-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:09.616762 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:09.616770    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxrhk\" (UniqueName: \"kubernetes.io/projected/089b836d-7fcf-4210-9259-856c8d8fde41-kube-api-access-hxrhk\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:10.188131 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.188092    4420 generic.go:356] "Generic (PLEG): container finished" podID="089b836d-7fcf-4210-9259-856c8d8fde41" containerID="2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560" exitCode=0
Oct 30 19:59:10.188601 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.188136    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"
Oct 30 19:59:10.188601 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.188141    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w" event={"ID":"089b836d-7fcf-4210-9259-856c8d8fde41","Type":"ContainerDied","Data":"2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560"}
Oct 30 19:59:10.188601 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.188221    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w" event={"ID":"089b836d-7fcf-4210-9259-856c8d8fde41","Type":"ContainerDied","Data":"fbd67e64143ede7e19cf1d2a928820d6d7947051443c2855e1eb1311f00874d6"}
Oct 30 19:59:10.188601 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.188255    4420 scope.go:122] "RemoveContainer" containerID="2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560"
Oct 30 19:59:10.198973 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.198958    4420 scope.go:122] "RemoveContainer" containerID="2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560"
Oct 30 19:59:10.199225 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:10.199208    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560\": container with ID starting with 2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560 not found: ID does not exist" containerID="2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560"
Oct 30 19:59:10.199274 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.199235    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560"} err="failed to get container status \"2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560\": rpc error: code = NotFound desc = could not find container \"2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560\": container with ID starting with 2de2b1bc1c9224b602ecb25bea860e280f0314991da40001f8f5ae84c39c0560 not found: ID does not exist"
Oct 30 19:59:10.353042 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.353015    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"]
Oct 30 19:59:10.368976 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.368948    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-75b38d-7fdf5b5665-gsg5w"]
Oct 30 19:59:10.430523 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.430468    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"]
Oct 30 19:59:10.430700 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.430678    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv" podUID="8d3ad8c3-3408-4e6b-b728-450d3f14466d" containerName="maas-api" containerID="cri-o://21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee" gracePeriod=30
Oct 30 19:59:10.715302 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.715276    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:59:10.837108 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.837067    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d3ad8c3-3408-4e6b-b728-450d3f14466d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ad8c3-3408-4e6b-b728-450d3f14466d-openshift-service-ca\") pod \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") "
Oct 30 19:59:10.837453 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.837425    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3ad8c3-3408-4e6b-b728-450d3f14466d-kube-api-access-xzxzx\" (UniqueName: \"kubernetes.io/projected/8d3ad8c3-3408-4e6b-b728-450d3f14466d-kube-api-access-xzxzx\") pod \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") "
Oct 30 19:59:10.837582 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.837540    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-maas-api-tls\") pod \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") "
Oct 30 19:59:10.837687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.837672    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-metrics-certs\") pod \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\" (UID: \"8d3ad8c3-3408-4e6b-b728-450d3f14466d\") "
Oct 30 19:59:10.845406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.841570    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-metrics-certs" pod "8d3ad8c3-3408-4e6b-b728-450d3f14466d" (UID: "8d3ad8c3-3408-4e6b-b728-450d3f14466d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:10.845406 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.845366    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3ad8c3-3408-4e6b-b728-450d3f14466d-openshift-service-ca" pod "8d3ad8c3-3408-4e6b-b728-450d3f14466d" (UID: "8d3ad8c3-3408-4e6b-b728-450d3f14466d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:10.856963 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.856913    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3ad8c3-3408-4e6b-b728-450d3f14466d-kube-api-access-xzxzx" pod "8d3ad8c3-3408-4e6b-b728-450d3f14466d" (UID: "8d3ad8c3-3408-4e6b-b728-450d3f14466d"). InnerVolumeSpecName "kube-api-access-xzxzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:10.857574 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.857532    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-maas-api-tls" pod "8d3ad8c3-3408-4e6b-b728-450d3f14466d" (UID: "8d3ad8c3-3408-4e6b-b728-450d3f14466d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:10.899928 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.899900    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089b836d-7fcf-4210-9259-856c8d8fde41" path="/var/lib/kubelet/pods/089b836d-7fcf-4210-9259-856c8d8fde41/volumes"
Oct 30 19:59:10.940208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.940153    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:10.940208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.940176    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d3ad8c3-3408-4e6b-b728-450d3f14466d-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:10.940208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.940185    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzxzx\" (UniqueName: \"kubernetes.io/projected/8d3ad8c3-3408-4e6b-b728-450d3f14466d-kube-api-access-xzxzx\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:10.940208 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:10.940194    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d3ad8c3-3408-4e6b-b728-450d3f14466d-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:11.197040 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:11.196967    4420 generic.go:356] "Generic (PLEG): container finished" podID="8d3ad8c3-3408-4e6b-b728-450d3f14466d" containerID="21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee" exitCode=0
Oct 30 19:59:11.197040 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:11.197017    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"
Oct 30 19:59:11.197507 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:11.197051    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv" event={"ID":"8d3ad8c3-3408-4e6b-b728-450d3f14466d","Type":"ContainerDied","Data":"21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee"}
Oct 30 19:59:11.197507 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:11.197101    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv" event={"ID":"8d3ad8c3-3408-4e6b-b728-450d3f14466d","Type":"ContainerDied","Data":"a7f276a6a5f0c5c67c15776e23f5deb7d768cb0f4e3ae7f41df8b585bdee587d"}
Oct 30 19:59:11.197507 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:11.197123    4420 scope.go:122] "RemoveContainer" containerID="21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee"
Oct 30 19:59:11.214121 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:11.214092    4420 scope.go:122] "RemoveContainer" containerID="21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee"
Oct 30 19:59:11.214500 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:11.214465    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee\": container with ID starting with 21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee not found: ID does not exist" containerID="21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee"
Oct 30 19:59:11.214576 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:11.214509    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee"} err="failed to get container status \"21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee\": rpc error: code = NotFound desc = could not find container \"21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee\": container with ID starting with 21c1f31c76287578c07041e5d860be32d0a0eeb023f7b8f8a9c071d3f0a6f1ee not found: ID does not exist"
Oct 30 19:59:11.364158 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:11.364132    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"]
Oct 30 19:59:11.366783 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:11.366764    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb14ed-f8c84545c-qjzkv"]
Oct 30 19:59:12.420463 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.420401    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:59:12.462367 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462339    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-socket\") pod \"ac27ddeb-8cb2-473a-b036-3f0201411622\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") "
Oct 30 19:59:12.462492 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462386    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-data\") pod \"ac27ddeb-8cb2-473a-b036-3f0201411622\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") "
Oct 30 19:59:12.462492 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462416    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-certs\") pod \"ac27ddeb-8cb2-473a-b036-3f0201411622\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") "
Oct 30 19:59:12.462578 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462529    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac27ddeb-8cb2-473a-b036-3f0201411622-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac27ddeb-8cb2-473a-b036-3f0201411622-istiod-ca-cert\") pod \"ac27ddeb-8cb2-473a-b036-3f0201411622\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") "
Oct 30 19:59:12.462633 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462577    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-envoy\") pod \"ac27ddeb-8cb2-473a-b036-3f0201411622\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") "
Oct 30 19:59:12.462633 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462606    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-socket" pod "ac27ddeb-8cb2-473a-b036-3f0201411622" (UID: "ac27ddeb-8cb2-473a-b036-3f0201411622"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:12.462743 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462648    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-data" pod "ac27ddeb-8cb2-473a-b036-3f0201411622" (UID: "ac27ddeb-8cb2-473a-b036-3f0201411622"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:12.462743 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462732    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-certs" pod "ac27ddeb-8cb2-473a-b036-3f0201411622" (UID: "ac27ddeb-8cb2-473a-b036-3f0201411622"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:12.462905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462749    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-token\" (UniqueName: \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-token\") pod \"ac27ddeb-8cb2-473a-b036-3f0201411622\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") "
Oct 30 19:59:12.462905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462780    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-podinfo\") pod \"ac27ddeb-8cb2-473a-b036-3f0201411622\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") "
Oct 30 19:59:12.462905 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462883    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-kube-api-access-ts42p\" (UniqueName: \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-kube-api-access-ts42p\") pod \"ac27ddeb-8cb2-473a-b036-3f0201411622\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") "
Oct 30 19:59:12.463052 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462908    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-credential-socket\") pod \"ac27ddeb-8cb2-473a-b036-3f0201411622\" (UID: \"ac27ddeb-8cb2-473a-b036-3f0201411622\") "
Oct 30 19:59:12.463052 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.462945    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac27ddeb-8cb2-473a-b036-3f0201411622-istiod-ca-cert" pod "ac27ddeb-8cb2-473a-b036-3f0201411622" (UID: "ac27ddeb-8cb2-473a-b036-3f0201411622"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:12.463288 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.463205    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-credential-socket" pod "ac27ddeb-8cb2-473a-b036-3f0201411622" (UID: "ac27ddeb-8cb2-473a-b036-3f0201411622"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:12.463638 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.463615    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:12.463747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.463643    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:12.463747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.463662    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:12.463747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.463675    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:12.463747 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.463703    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac27ddeb-8cb2-473a-b036-3f0201411622-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:12.464909 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.464888    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-envoy" pod "ac27ddeb-8cb2-473a-b036-3f0201411622" (UID: "ac27ddeb-8cb2-473a-b036-3f0201411622"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:12.465031 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.465017    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-token" pod "ac27ddeb-8cb2-473a-b036-3f0201411622" (UID: "ac27ddeb-8cb2-473a-b036-3f0201411622"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:12.465432 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.465412    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-kube-api-access-ts42p" pod "ac27ddeb-8cb2-473a-b036-3f0201411622" (UID: "ac27ddeb-8cb2-473a-b036-3f0201411622"). InnerVolumeSpecName "kube-api-access-ts42p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:12.465432 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.465423    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-podinfo" pod "ac27ddeb-8cb2-473a-b036-3f0201411622" (UID: "ac27ddeb-8cb2-473a-b036-3f0201411622"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:12.565449 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.565418    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:12.565449 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.565446    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:12.565449 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.565455    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts42p\" (UniqueName: \"kubernetes.io/projected/ac27ddeb-8cb2-473a-b036-3f0201411622-kube-api-access-ts42p\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:12.565692 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.565463    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac27ddeb-8cb2-473a-b036-3f0201411622-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:12.897740 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:12.897710    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d3ad8c3-3408-4e6b-b728-450d3f14466d" path="/var/lib/kubelet/pods/8d3ad8c3-3408-4e6b-b728-450d3f14466d/volumes"
Oct 30 19:59:13.212329 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.212257    4420 generic.go:356] "Generic (PLEG): container finished" podID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerID="81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7" exitCode=0
Oct 30 19:59:13.212329 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.212304    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"
Oct 30 19:59:13.212329 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.212324    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" event={"ID":"ac27ddeb-8cb2-473a-b036-3f0201411622","Type":"ContainerDied","Data":"81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7"}
Oct 30 19:59:13.212590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.212371    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f" event={"ID":"ac27ddeb-8cb2-473a-b036-3f0201411622","Type":"ContainerDied","Data":"cbbc60820adb9e99b22ddca48023e2ebf2ccbec4488f4334d61a462c14d8dfa0"}
Oct 30 19:59:13.212590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.212393    4420 scope.go:122] "RemoveContainer" containerID="81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7"
Oct 30 19:59:13.222101 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.222082    4420 scope.go:122] "RemoveContainer" containerID="a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1"
Oct 30 19:59:13.231195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.231170    4420 scope.go:122] "RemoveContainer" containerID="81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7"
Oct 30 19:59:13.231483 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:13.231465    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7\": container with ID starting with 81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7 not found: ID does not exist" containerID="81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7"
Oct 30 19:59:13.231565 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.231491    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7"} err="failed to get container status \"81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7\": rpc error: code = NotFound desc = could not find container \"81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7\": container with ID starting with 81d580ea9d1e75bb98ee04197eeba1cffdc4a303b2e822388473db9ff677b5e7 not found: ID does not exist"
Oct 30 19:59:13.231565 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.231512    4420 scope.go:122] "RemoveContainer" containerID="a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1"
Oct 30 19:59:13.231788 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:13.231769    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1\": container with ID starting with a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1 not found: ID does not exist" containerID="a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1"
Oct 30 19:59:13.231902 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.231792    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1"} err="failed to get container status \"a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1\": rpc error: code = NotFound desc = could not find container \"a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1\": container with ID starting with a4b88cc57551cd123820efd6593de54a279a061f28a13a13d79983a159679af1 not found: ID does not exist"
Oct 30 19:59:13.383829 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.383792    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"]
Oct 30 19:59:13.433687 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:13.433663    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-86e4a7-openshift-default-67b7c7f57f-rbf2f"]
Oct 30 19:59:14.897574 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:14.897541    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" path="/var/lib/kubelet/pods/ac27ddeb-8cb2-473a-b036-3f0201411622/volumes"
Oct 30 19:59:21.719764 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:21.719735    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"]
Oct 30 19:59:21.720233 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:21.719954    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy" containerID="cri-o://b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead" gracePeriod=30
Oct 30 19:59:26.992490 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:26.992436    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:59:27.131855 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.131827    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-podinfo\") pod \"b74aa3ef-91cc-4ece-b74e-c3332572e110\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") "
Oct 30 19:59:27.132005 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.131900    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-envoy\") pod \"b74aa3ef-91cc-4ece-b74e-c3332572e110\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") "
Oct 30 19:59:27.132005 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.131964    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-certs\") pod \"b74aa3ef-91cc-4ece-b74e-c3332572e110\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") "
Oct 30 19:59:27.132127 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.132003    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74aa3ef-91cc-4ece-b74e-c3332572e110-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74aa3ef-91cc-4ece-b74e-c3332572e110-istiod-ca-cert\") pod \"b74aa3ef-91cc-4ece-b74e-c3332572e110\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") "
Oct 30 19:59:27.132127 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.132025    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-credential-socket\") pod \"b74aa3ef-91cc-4ece-b74e-c3332572e110\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") "
Oct 30 19:59:27.132127 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.132068    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-token\" (UniqueName: \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-token\") pod \"b74aa3ef-91cc-4ece-b74e-c3332572e110\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") "
Oct 30 19:59:27.132127 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.132092    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-socket\") pod \"b74aa3ef-91cc-4ece-b74e-c3332572e110\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") "
Oct 30 19:59:27.132341 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.132203    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-kube-api-access-xq69q\" (UniqueName: \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-kube-api-access-xq69q\") pod \"b74aa3ef-91cc-4ece-b74e-c3332572e110\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") "
Oct 30 19:59:27.132399 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.132351    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-data\") pod \"b74aa3ef-91cc-4ece-b74e-c3332572e110\" (UID: \"b74aa3ef-91cc-4ece-b74e-c3332572e110\") "
Oct 30 19:59:27.132665 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.132638    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-credential-socket" pod "b74aa3ef-91cc-4ece-b74e-c3332572e110" (UID: "b74aa3ef-91cc-4ece-b74e-c3332572e110"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:27.133004 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.132976    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-socket" pod "b74aa3ef-91cc-4ece-b74e-c3332572e110" (UID: "b74aa3ef-91cc-4ece-b74e-c3332572e110"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:27.133004 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.132988    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-data" pod "b74aa3ef-91cc-4ece-b74e-c3332572e110" (UID: "b74aa3ef-91cc-4ece-b74e-c3332572e110"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:27.133219 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.133154    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74aa3ef-91cc-4ece-b74e-c3332572e110-istiod-ca-cert" pod "b74aa3ef-91cc-4ece-b74e-c3332572e110" (UID: "b74aa3ef-91cc-4ece-b74e-c3332572e110"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:27.133557 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.133514    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-certs" pod "b74aa3ef-91cc-4ece-b74e-c3332572e110" (UID: "b74aa3ef-91cc-4ece-b74e-c3332572e110"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:27.134452 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.134429    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-podinfo" pod "b74aa3ef-91cc-4ece-b74e-c3332572e110" (UID: "b74aa3ef-91cc-4ece-b74e-c3332572e110"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:27.134983 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.134954    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-envoy" pod "b74aa3ef-91cc-4ece-b74e-c3332572e110" (UID: "b74aa3ef-91cc-4ece-b74e-c3332572e110"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:27.135064 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.134959    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-token" pod "b74aa3ef-91cc-4ece-b74e-c3332572e110" (UID: "b74aa3ef-91cc-4ece-b74e-c3332572e110"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:27.135425 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.135405    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-kube-api-access-xq69q" pod "b74aa3ef-91cc-4ece-b74e-c3332572e110" (UID: "b74aa3ef-91cc-4ece-b74e-c3332572e110"). InnerVolumeSpecName "kube-api-access-xq69q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:27.234309 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.234284    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:27.234309 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.234308    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:27.234538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.234317    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:27.234538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.234330    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:27.234538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.234344    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74aa3ef-91cc-4ece-b74e-c3332572e110-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:27.234538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.234353    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:27.234538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.234362    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:27.234538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.234370    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74aa3ef-91cc-4ece-b74e-c3332572e110-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:27.234538 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.234379    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq69q\" (UniqueName: \"kubernetes.io/projected/b74aa3ef-91cc-4ece-b74e-c3332572e110-kube-api-access-xq69q\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:27.309739 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.309669    4420 generic.go:356] "Generic (PLEG): container finished" podID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerID="b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead" exitCode=0
Oct 30 19:59:27.309739 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.309719    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"
Oct 30 19:59:27.309935 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.309760    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" event={"ID":"b74aa3ef-91cc-4ece-b74e-c3332572e110","Type":"ContainerDied","Data":"b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead"}
Oct 30 19:59:27.309935 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.309841    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895" event={"ID":"b74aa3ef-91cc-4ece-b74e-c3332572e110","Type":"ContainerDied","Data":"2c96c7dc4e949c14d2b0b0f6cc50fbdb5ce71b34f47ebd9b51000e698032eec9"}
Oct 30 19:59:27.309935 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.309872    4420 scope.go:122] "RemoveContainer" containerID="b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead"
Oct 30 19:59:27.320664 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.320647    4420 scope.go:122] "RemoveContainer" containerID="502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19"
Oct 30 19:59:27.329611 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.329594    4420 scope.go:122] "RemoveContainer" containerID="b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead"
Oct 30 19:59:27.329883 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:27.329860    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead\": container with ID starting with b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead not found: ID does not exist" containerID="b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead"
Oct 30 19:59:27.329951 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.329895    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead"} err="failed to get container status \"b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead\": rpc error: code = NotFound desc = could not find container \"b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead\": container with ID starting with b74b77f8d8b827c8dc437a65e30af3d8a8c4f8f67546821fb776e6257e43aead not found: ID does not exist"
Oct 30 19:59:27.329951 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.329918    4420 scope.go:122] "RemoveContainer" containerID="502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19"
Oct 30 19:59:27.330169 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:27.330152    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19\": container with ID starting with 502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19 not found: ID does not exist" containerID="502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19"
Oct 30 19:59:27.330220 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.330175    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19"} err="failed to get container status \"502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19\": rpc error: code = NotFound desc = could not find container \"502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19\": container with ID starting with 502513dce2c8a7c7da8f36398c0ebbe7ea9339a9321a7d7d180870d73d646b19 not found: ID does not exist"
Oct 30 19:59:27.476407 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.476375    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"]
Oct 30 19:59:27.482771 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:27.482751    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cb14ed-openshift-default-68f97495c9-68895"]
Oct 30 19:59:28.897481 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:28.897444    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" path="/var/lib/kubelet/pods/b74aa3ef-91cc-4ece-b74e-c3332572e110/volumes"
Oct 30 19:59:35.197242 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:35.197207    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 19:59:48.459784 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.459750    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"]
Oct 30 19:59:48.460170 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.459954    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc" podUID="db8bc9fc-79eb-466f-9f07-35f37e2798d4" containerName="maas-api" containerID="cri-o://dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235" gracePeriod=30
Oct 30 19:59:48.745479 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.745456    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:59:48.766993 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.766964    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-maas-api-tls\") pod \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") "
Oct 30 19:59:48.767100 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.767010    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-metrics-certs\") pod \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") "
Oct 30 19:59:48.767100 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.767051    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db8bc9fc-79eb-466f-9f07-35f37e2798d4-kube-api-access-hhxcb\" (UniqueName: \"kubernetes.io/projected/db8bc9fc-79eb-466f-9f07-35f37e2798d4-kube-api-access-hhxcb\") pod \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") "
Oct 30 19:59:48.767210 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.767106    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db8bc9fc-79eb-466f-9f07-35f37e2798d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db8bc9fc-79eb-466f-9f07-35f37e2798d4-openshift-service-ca\") pod \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\" (UID: \"db8bc9fc-79eb-466f-9f07-35f37e2798d4\") "
Oct 30 19:59:48.767516 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.767486    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db8bc9fc-79eb-466f-9f07-35f37e2798d4-openshift-service-ca" pod "db8bc9fc-79eb-466f-9f07-35f37e2798d4" (UID: "db8bc9fc-79eb-466f-9f07-35f37e2798d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:48.767774 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.767580    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db8bc9fc-79eb-466f-9f07-35f37e2798d4-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:48.769050 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.769025    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-maas-api-tls" pod "db8bc9fc-79eb-466f-9f07-35f37e2798d4" (UID: "db8bc9fc-79eb-466f-9f07-35f37e2798d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:48.769297 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.769268    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db8bc9fc-79eb-466f-9f07-35f37e2798d4-kube-api-access-hhxcb" pod "db8bc9fc-79eb-466f-9f07-35f37e2798d4" (UID: "db8bc9fc-79eb-466f-9f07-35f37e2798d4"). InnerVolumeSpecName "kube-api-access-hhxcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:48.769587 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.769573    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-metrics-certs" pod "db8bc9fc-79eb-466f-9f07-35f37e2798d4" (UID: "db8bc9fc-79eb-466f-9f07-35f37e2798d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:48.868580 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.868549    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:48.868580 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.868578    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db8bc9fc-79eb-466f-9f07-35f37e2798d4-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:48.868779 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:48.868590    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhxcb\" (UniqueName: \"kubernetes.io/projected/db8bc9fc-79eb-466f-9f07-35f37e2798d4-kube-api-access-hhxcb\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:49.471076 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.471039    4420 generic.go:356] "Generic (PLEG): container finished" podID="db8bc9fc-79eb-466f-9f07-35f37e2798d4" containerID="dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235" exitCode=0
Oct 30 19:59:49.471695 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.471209    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"
Oct 30 19:59:49.471922 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.471230    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc" event={"ID":"db8bc9fc-79eb-466f-9f07-35f37e2798d4","Type":"ContainerDied","Data":"dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235"}
Oct 30 19:59:49.472072 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.472054    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc" event={"ID":"db8bc9fc-79eb-466f-9f07-35f37e2798d4","Type":"ContainerDied","Data":"a658fd3e2de65460c1ec464b1719d4187800a4d0e03f306bdf9d6c7973b2cfc3"}
Oct 30 19:59:49.472195 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.472181    4420 scope.go:122] "RemoveContainer" containerID="dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235"
Oct 30 19:59:49.487030 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.486978    4420 scope.go:122] "RemoveContainer" containerID="dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235"
Oct 30 19:59:49.487444 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:49.487423    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235\": container with ID starting with dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235 not found: ID does not exist" containerID="dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235"
Oct 30 19:59:49.487536 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.487458    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235"} err="failed to get container status \"dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235\": rpc error: code = NotFound desc = could not find container \"dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235\": container with ID starting with dfb62261afba9c3e777d3a80cf818adcd90a3bfd68ab4f5118f4541cdf80c235 not found: ID does not exist"
Oct 30 19:59:49.637365 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.637339    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"]
Oct 30 19:59:49.640152 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.640133    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b0c827-8d8dd9bd4-lpggc"]
Oct 30 19:59:49.685789 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.685764    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"]
Oct 30 19:59:49.686022 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.685994    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth" podUID="a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7" containerName="maas-api" containerID="cri-o://e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0" gracePeriod=30
Oct 30 19:59:49.946502 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.946480    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:59:49.983750 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.983719    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-maas-api-tls\") pod \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") "
Oct 30 19:59:49.983922 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.983789    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-metrics-certs\") pod \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") "
Oct 30 19:59:49.984055 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.984039    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-openshift-service-ca\") pod \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") "
Oct 30 19:59:49.984129 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.984112    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-kube-api-access-krbgr\" (UniqueName: \"kubernetes.io/projected/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-kube-api-access-krbgr\") pod \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\" (UID: \"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7\") "
Oct 30 19:59:49.984575 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.984545    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-openshift-service-ca" pod "a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7" (UID: "a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:49.984723 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.984705    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:49.985944 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.985921    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-maas-api-tls" pod "a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7" (UID: "a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:49.986063 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.985955    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-metrics-certs" pod "a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7" (UID: "a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:49.986116 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:49.986073    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-kube-api-access-krbgr" pod "a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7" (UID: "a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7"). InnerVolumeSpecName "kube-api-access-krbgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:50.086101 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.086011    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krbgr\" (UniqueName: \"kubernetes.io/projected/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-kube-api-access-krbgr\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:50.086101 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.086061    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:50.086101 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.086076    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:50.478844 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.478800    4420 generic.go:356] "Generic (PLEG): container finished" podID="a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7" containerID="e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0" exitCode=0
Oct 30 19:59:50.479302 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.478871    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"
Oct 30 19:59:50.479302 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.478890    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth" event={"ID":"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7","Type":"ContainerDied","Data":"e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0"}
Oct 30 19:59:50.479302 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.478977    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth" event={"ID":"a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7","Type":"ContainerDied","Data":"19961a1d9483be0717b4e3f775de5e13766e74c4be674b70d595721defe955e1"}
Oct 30 19:59:50.479302 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.479007    4420 scope.go:122] "RemoveContainer" containerID="e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0"
Oct 30 19:59:50.489155 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.489139    4420 scope.go:122] "RemoveContainer" containerID="e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0"
Oct 30 19:59:50.489396 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:50.489379    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0\": container with ID starting with e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0 not found: ID does not exist" containerID="e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0"
Oct 30 19:59:50.489443 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.489405    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0"} err="failed to get container status \"e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0\": rpc error: code = NotFound desc = could not find container \"e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0\": container with ID starting with e6dabd0ca8f8cce17ae55d39b4aa6c622c174d82f9e34b922202b81a88e97cd0 not found: ID does not exist"
Oct 30 19:59:50.645039 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.645021    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"]
Oct 30 19:59:50.650312 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.650289    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-2e5977-66bdcb94dd-rrjth"]
Oct 30 19:59:50.899273 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.899229    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7" path="/var/lib/kubelet/pods/a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7/volumes"
Oct 30 19:59:50.900034 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:50.900012    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db8bc9fc-79eb-466f-9f07-35f37e2798d4" path="/var/lib/kubelet/pods/db8bc9fc-79eb-466f-9f07-35f37e2798d4/volumes"
Oct 30 19:59:51.782250 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:51.782220    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"]
Oct 30 19:59:51.782694 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:51.782451    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy" containerID="cri-o://d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1" gracePeriod=30
Oct 30 19:59:57.047727 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.047677    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:59:57.067895 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.067869    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-podinfo\") pod \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") "
Oct 30 19:59:57.067994 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.067902    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-credential-socket\") pod \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") "
Oct 30 19:59:57.067994 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.067953    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-kube-api-access-zjqjp\" (UniqueName: \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-kube-api-access-zjqjp\") pod \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") "
Oct 30 19:59:57.068067 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.068014    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-data\") pod \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") "
Oct 30 19:59:57.068128 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.068114    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-certs\") pod \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") "
Oct 30 19:59:57.068206 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.068194    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-envoy\") pod \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") "
Oct 30 19:59:57.068255 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.068234    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-socket\") pod \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") "
Oct 30 19:59:57.068314 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.068258    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-credential-socket" pod "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" (UID: "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:57.068314 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.068271    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-data" pod "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" (UID: "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:57.068404 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.068322    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istiod-ca-cert\") pod \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") "
Oct 30 19:59:57.068404 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.068380    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-token\" (UniqueName: \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-token\") pod \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\" (UID: \"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e\") "
Oct 30 19:59:57.068404 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.068377    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-certs" pod "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" (UID: "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:57.068703 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.068498    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-socket" pod "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" (UID: "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:57.068874 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.068851    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istiod-ca-cert" pod "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" (UID: "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:57.069097 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.069078    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.069165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.069104    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.069165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.069119    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.069165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.069132    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.069165 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.069149    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.070271 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.070244    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-podinfo" pod "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" (UID: "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:59:57.070271 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.070250    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-kube-api-access-zjqjp" pod "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" (UID: "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e"). InnerVolumeSpecName "kube-api-access-zjqjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:57.070419 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.070294    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-envoy" pod "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" (UID: "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:59:57.070419 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.070404    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-token" pod "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" (UID: "ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:57.170752 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.170726    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.170752 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.170750    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.170931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.170762    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjqjp\" (UniqueName: \"kubernetes.io/projected/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-kube-api-access-zjqjp\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.170931 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.170771    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.292704 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.292676    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"]
Oct 30 19:59:57.523654 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.523621    4420 generic.go:356] "Generic (PLEG): container finished" podID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerID="d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1" exitCode=0
Oct 30 19:59:57.523850 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.523681    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"
Oct 30 19:59:57.523850 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.523700    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" event={"ID":"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e","Type":"ContainerDied","Data":"d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1"}
Oct 30 19:59:57.523850 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.523752    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87" event={"ID":"ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e","Type":"ContainerDied","Data":"8a9302865a7c3cb2967b7ee310c1cc435360976ac819e807ad75b3c246623006"}
Oct 30 19:59:57.523850 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.523773    4420 scope.go:122] "RemoveContainer" containerID="d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1"
Oct 30 19:59:57.534407 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.534392    4420 scope.go:122] "RemoveContainer" containerID="7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe"
Oct 30 19:59:57.544267 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.544246    4420 scope.go:122] "RemoveContainer" containerID="d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1"
Oct 30 19:59:57.544534 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:57.544516    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1\": container with ID starting with d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1 not found: ID does not exist" containerID="d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1"
Oct 30 19:59:57.544590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.544542    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1"} err="failed to get container status \"d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1\": rpc error: code = NotFound desc = could not find container \"d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1\": container with ID starting with d6b88fb2b6eb0b6adad70c9e685d7fb3fbacb49f27c3b38a0a90ac75e94ff1f1 not found: ID does not exist"
Oct 30 19:59:57.544590 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.544563    4420 scope.go:122] "RemoveContainer" containerID="7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe"
Oct 30 19:59:57.544805 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:57.544790    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe\": container with ID starting with 7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe not found: ID does not exist" containerID="7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe"
Oct 30 19:59:57.544883 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.544845    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe"} err="failed to get container status \"7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe\": rpc error: code = NotFound desc = could not find container \"7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe\": container with ID starting with 7db38e8fe64a5ef3968dc38552b7ccbed8c0fe19b63fe097fbcc30a805cc2bbe not found: ID does not exist"
Oct 30 19:59:57.588240 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.588219    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"]
Oct 30 19:59:57.588387 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.588372    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p" podUID="2c081125-76ff-454c-80f5-2f06fe73dbd8" containerName="maas-api" containerID="cri-o://982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0" gracePeriod=30
Oct 30 19:59:57.689900 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.689868    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"]
Oct 30 19:59:57.696303 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.696275    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-da4c62-openshift-default-5945967dc-zls87"]
Oct 30 19:59:57.855356 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.855322    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:59:57.879587 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.879563    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c081125-76ff-454c-80f5-2f06fe73dbd8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c081125-76ff-454c-80f5-2f06fe73dbd8-openshift-service-ca\") pod \"2c081125-76ff-454c-80f5-2f06fe73dbd8\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") "
Oct 30 19:59:57.879696 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.879620    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c081125-76ff-454c-80f5-2f06fe73dbd8-kube-api-access-2d8rk\" (UniqueName: \"kubernetes.io/projected/2c081125-76ff-454c-80f5-2f06fe73dbd8-kube-api-access-2d8rk\") pod \"2c081125-76ff-454c-80f5-2f06fe73dbd8\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") "
Oct 30 19:59:57.879732 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.879704    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-maas-api-tls\") pod \"2c081125-76ff-454c-80f5-2f06fe73dbd8\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") "
Oct 30 19:59:57.879787 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.879771    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-metrics-certs\") pod \"2c081125-76ff-454c-80f5-2f06fe73dbd8\" (UID: \"2c081125-76ff-454c-80f5-2f06fe73dbd8\") "
Oct 30 19:59:57.879912 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.879890    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c081125-76ff-454c-80f5-2f06fe73dbd8-openshift-service-ca" pod "2c081125-76ff-454c-80f5-2f06fe73dbd8" (UID: "2c081125-76ff-454c-80f5-2f06fe73dbd8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:59:57.880202 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.880183    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c081125-76ff-454c-80f5-2f06fe73dbd8-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.881908 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.881866    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-maas-api-tls" pod "2c081125-76ff-454c-80f5-2f06fe73dbd8" (UID: "2c081125-76ff-454c-80f5-2f06fe73dbd8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:57.882017 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.881926    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-metrics-certs" pod "2c081125-76ff-454c-80f5-2f06fe73dbd8" (UID: "2c081125-76ff-454c-80f5-2f06fe73dbd8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:59:57.882059 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.882035    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c081125-76ff-454c-80f5-2f06fe73dbd8-kube-api-access-2d8rk" pod "2c081125-76ff-454c-80f5-2f06fe73dbd8" (UID: "2c081125-76ff-454c-80f5-2f06fe73dbd8"). InnerVolumeSpecName "kube-api-access-2d8rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:59:57.981767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.981741    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.981767 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.981766    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d8rk\" (UniqueName: \"kubernetes.io/projected/2c081125-76ff-454c-80f5-2f06fe73dbd8-kube-api-access-2d8rk\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:57.982015 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:57.981778    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c081125-76ff-454c-80f5-2f06fe73dbd8-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 19:59:58.566892 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:58.566851    4420 generic.go:356] "Generic (PLEG): container finished" podID="2c081125-76ff-454c-80f5-2f06fe73dbd8" containerID="982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0" exitCode=0
Oct 30 19:59:58.567627 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:58.566912    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p" event={"ID":"2c081125-76ff-454c-80f5-2f06fe73dbd8","Type":"ContainerDied","Data":"982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0"}
Oct 30 19:59:58.568358 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:58.567694    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p" event={"ID":"2c081125-76ff-454c-80f5-2f06fe73dbd8","Type":"ContainerDied","Data":"adbcbd8dfa08594f741ba64842777b1497c927d786c2be95550df4a5dca19152"}
Oct 30 19:59:58.568541 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:58.567870    4420 scope.go:122] "RemoveContainer" containerID="982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0"
Oct 30 19:59:58.568713 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:58.567005    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"
Oct 30 19:59:58.588683 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:58.588661    4420 scope.go:122] "RemoveContainer" containerID="982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0"
Oct 30 19:59:58.589024 ip-10-0-133-7 kubenswrapper[4420]: E1030 19:59:58.589003    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0\": container with ID starting with 982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0 not found: ID does not exist" containerID="982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0"
Oct 30 19:59:58.589088 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:58.589033    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0"} err="failed to get container status \"982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0\": rpc error: code = NotFound desc = could not find container \"982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0\": container with ID starting with 982e957033d80f0426593191ad2d504cf58bb9b8c35da51244aac8288b6bc8d0 not found: ID does not exist"
Oct 30 19:59:58.740505 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:58.740475    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"]
Oct 30 19:59:58.789551 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:58.789529    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-dd9b8d79-6bb7bd58-pr62p"]
Oct 30 19:59:58.896604 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:58.896578    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c081125-76ff-454c-80f5-2f06fe73dbd8" path="/var/lib/kubelet/pods/2c081125-76ff-454c-80f5-2f06fe73dbd8/volumes"
Oct 30 19:59:58.897033 ip-10-0-133-7 kubenswrapper[4420]: I1030 19:59:58.897020    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" path="/var/lib/kubelet/pods/ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e/volumes"
Oct 30 20:00:00.444596 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.444566    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"]
Oct 30 20:00:00.445074 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.444792    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy" containerID="cri-o://faf7011b49cd8743c04e227ef5e87e95b96464589ccc4c12bf947fbca6c3474a" gracePeriod=30
Oct 30 20:00:00.857356 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857283    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"]
Oct 30 20:00:00.857805 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857793    4420 cpu_manager.go:414] "containerMap: removing container" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy"
Oct 30 20:00:00.857882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857807    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy"
Oct 30 20:00:00.857882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857830    4420 cpu_manager.go:414] "containerMap: removing container" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy"
Oct 30 20:00:00.857882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857835    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy"
Oct 30 20:00:00.857882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857846    4420 cpu_manager.go:414] "containerMap: removing container" podUID="089b836d-7fcf-4210-9259-856c8d8fde41" containerName="maas-api"
Oct 30 20:00:00.857882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857852    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089b836d-7fcf-4210-9259-856c8d8fde41" containerName="maas-api"
Oct 30 20:00:00.857882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857865    4420 cpu_manager.go:414] "containerMap: removing container" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy"
Oct 30 20:00:00.857882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857869    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy"
Oct 30 20:00:00.857882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857875    4420 cpu_manager.go:414] "containerMap: removing container" podUID="2c081125-76ff-454c-80f5-2f06fe73dbd8" containerName="maas-api"
Oct 30 20:00:00.857882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857880    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c081125-76ff-454c-80f5-2f06fe73dbd8" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857888    4420 cpu_manager.go:414] "containerMap: removing container" podUID="8d3ad8c3-3408-4e6b-b728-450d3f14466d" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857892    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3ad8c3-3408-4e6b-b728-450d3f14466d" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857900    4420 cpu_manager.go:414] "containerMap: removing container" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857906    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857917    4420 cpu_manager.go:414] "containerMap: removing container" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857921    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857927    4420 cpu_manager.go:414] "containerMap: removing container" podUID="db8bc9fc-79eb-466f-9f07-35f37e2798d4" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857931    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db8bc9fc-79eb-466f-9f07-35f37e2798d4" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857939    4420 cpu_manager.go:414] "containerMap: removing container" podUID="76483a35-32ab-46c0-9223-4ecfe57bff49" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857943    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76483a35-32ab-46c0-9223-4ecfe57bff49" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857955    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.857959    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858038    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a51a1ef9-c5e6-4a5e-ae69-edf73ea13fd7" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858047    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858052    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858059    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="089b836d-7fcf-4210-9259-856c8d8fde41" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858065    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858070    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c081125-76ff-454c-80f5-2f06fe73dbd8" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858075    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3ad8c3-3408-4e6b-b728-450d3f14466d" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858082    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac27ddeb-8cb2-473a-b036-3f0201411622" containerName="istio-proxy"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858088    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3c54ef-79b7-4c7a-bb9d-cdadfce5589e" containerName="istio-proxy"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858092    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="db8bc9fc-79eb-466f-9f07-35f37e2798d4" containerName="maas-api"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858096    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy"
Oct 30 20:00:00.858146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.858101    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="76483a35-32ab-46c0-9223-4ecfe57bff49" containerName="maas-api"
Oct 30 20:00:00.862778 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.862761    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:00.915345 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.915306    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:00:00.915455 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.915367    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:00:00.920368 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.920344    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:00.920582 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.920553    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:00.920756 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.920739    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7r4\" (UniqueName: \"kubernetes.io/projected/a26e766b-7314-455c-ae4f-0d7acca29891-kube-api-access-pd7r4\") pod \"maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:00.924850 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:00.924828    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"]
Oct 30 20:00:01.021604 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:01.021581    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7r4\" (UniqueName: \"kubernetes.io/projected/a26e766b-7314-455c-ae4f-0d7acca29891-kube-api-access-pd7r4\") pod \"maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:01.021734 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:01.021677    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:01.021734 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:01.021699    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:01.022390 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:01.022367    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:01.024099 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:01.024082    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:01.082387 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:01.082366    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7r4\" (UniqueName: \"kubernetes.io/projected/a26e766b-7314-455c-ae4f-0d7acca29891-kube-api-access-pd7r4\") pod \"maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:01.176785 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:01.176759    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:01.552341 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:01.552313    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"]
Oct 30 20:00:01.554014 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:00:01.553993    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26e766b_7314_455c_ae4f_0d7acca29891.slice/crio-a3bd69437a195b7898b9116c4cff90e7f065767cb90275a72a78a6bf4cfce0f6 WatchSource:0}: Error finding container a3bd69437a195b7898b9116c4cff90e7f065767cb90275a72a78a6bf4cfce0f6: Status 404 returned error can't find the container with id a3bd69437a195b7898b9116c4cff90e7f065767cb90275a72a78a6bf4cfce0f6
Oct 30 20:00:01.590859 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:01.590834    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd" event={"ID":"a26e766b-7314-455c-ae4f-0d7acca29891","Type":"ContainerStarted","Data":"a3bd69437a195b7898b9116c4cff90e7f065767cb90275a72a78a6bf4cfce0f6"}
Oct 30 20:00:02.598035 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:02.598005    4420 generic.go:356] "Generic (PLEG): container finished" podID="a26e766b-7314-455c-ae4f-0d7acca29891" containerID="056955c3e89779f0323ca8583770599eb944211b6a82eeddeebc39136007943f" exitCode=0
Oct 30 20:00:02.598442 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:02.598088    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd" event={"ID":"a26e766b-7314-455c-ae4f-0d7acca29891","Type":"ContainerDied","Data":"056955c3e89779f0323ca8583770599eb944211b6a82eeddeebc39136007943f"}
Oct 30 20:00:03.763661 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:03.763635    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:03.851186 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:03.851159    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26e766b-7314-455c-ae4f-0d7acca29891-kube-api-access-pd7r4\" (UniqueName: \"kubernetes.io/projected/a26e766b-7314-455c-ae4f-0d7acca29891-kube-api-access-pd7r4\") pod \"a26e766b-7314-455c-ae4f-0d7acca29891\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") "
Oct 30 20:00:03.851370 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:03.851352    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-xks-ca\") pod \"a26e766b-7314-455c-ae4f-0d7acca29891\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") "
Oct 30 20:00:03.851424 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:03.851401    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-openshift-ca\") pod \"a26e766b-7314-455c-ae4f-0d7acca29891\" (UID: \"a26e766b-7314-455c-ae4f-0d7acca29891\") "
Oct 30 20:00:03.851810 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:03.851778    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-openshift-ca" pod "a26e766b-7314-455c-ae4f-0d7acca29891" (UID: "a26e766b-7314-455c-ae4f-0d7acca29891"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:03.853269 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:03.853251    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26e766b-7314-455c-ae4f-0d7acca29891-kube-api-access-pd7r4" pod "a26e766b-7314-455c-ae4f-0d7acca29891" (UID: "a26e766b-7314-455c-ae4f-0d7acca29891"). InnerVolumeSpecName "kube-api-access-pd7r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:03.853674 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:03.853640    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-xks-ca" pod "a26e766b-7314-455c-ae4f-0d7acca29891" (UID: "a26e766b-7314-455c-ae4f-0d7acca29891"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:03.953087 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:03.953019    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:03.953087 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:03.953048    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a26e766b-7314-455c-ae4f-0d7acca29891-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:03.953087 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:03.953084    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd7r4\" (UniqueName: \"kubernetes.io/projected/a26e766b-7314-455c-ae4f-0d7acca29891-kube-api-access-pd7r4\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:04.610386 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:04.610310    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"
Oct 30 20:00:04.610386 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:04.610323    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd" event={"ID":"a26e766b-7314-455c-ae4f-0d7acca29891","Type":"ContainerDied","Data":"a3bd69437a195b7898b9116c4cff90e7f065767cb90275a72a78a6bf4cfce0f6"}
Oct 30 20:00:04.610386 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:04.610375    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3bd69437a195b7898b9116c4cff90e7f065767cb90275a72a78a6bf4cfce0f6"
Oct 30 20:00:04.650804 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:04.650778    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"]
Oct 30 20:00:05.196306 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:00:05.196276    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:00:05.619740 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.619703    4420 generic.go:356] "Generic (PLEG): container finished" podID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerID="faf7011b49cd8743c04e227ef5e87e95b96464589ccc4c12bf947fbca6c3474a" exitCode=0
Oct 30 20:00:05.619740 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.619734    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" event={"ID":"9cd22e64-3ad6-4f75-bc53-5dae3fccee49","Type":"ContainerDied","Data":"faf7011b49cd8743c04e227ef5e87e95b96464589ccc4c12bf947fbca6c3474a"}
Oct 30 20:00:05.619964 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.619797    4420 scope.go:122] "RemoveContainer" containerID="f1f7435d852f5be8e749468e4db038c213e716d7bdc4fd6d27a4c745b57c4af3"
Oct 30 20:00:05.734160 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.734131    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 20:00:05.881580 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.881554    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-podinfo\") pod \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") "
Oct 30 20:00:05.881700 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.881598    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-envoy\") pod \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") "
Oct 30 20:00:05.881700 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.881632    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-data\") pod \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") "
Oct 30 20:00:05.881700 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.881675    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-socket\") pod \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") "
Oct 30 20:00:05.881905 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.881721    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istiod-ca-cert\") pod \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") "
Oct 30 20:00:05.881905 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.881750    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-certs\") pod \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") "
Oct 30 20:00:05.881905 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.881774    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-credential-socket\") pod \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") "
Oct 30 20:00:05.881905 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.881869    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-kube-api-access-pflrq\" (UniqueName: \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-kube-api-access-pflrq\") pod \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") "
Oct 30 20:00:05.882107 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.881936    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-token\" (UniqueName: \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-token\") pod \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\" (UID: \"9cd22e64-3ad6-4f75-bc53-5dae3fccee49\") "
Oct 30 20:00:05.882107 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.881952    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-data" pod "9cd22e64-3ad6-4f75-bc53-5dae3fccee49" (UID: "9cd22e64-3ad6-4f75-bc53-5dae3fccee49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:05.882107 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.882026    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-socket" pod "9cd22e64-3ad6-4f75-bc53-5dae3fccee49" (UID: "9cd22e64-3ad6-4f75-bc53-5dae3fccee49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:05.882283 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.882264    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istiod-ca-cert" pod "9cd22e64-3ad6-4f75-bc53-5dae3fccee49" (UID: "9cd22e64-3ad6-4f75-bc53-5dae3fccee49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:05.882338 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.882282    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-credential-socket" pod "9cd22e64-3ad6-4f75-bc53-5dae3fccee49" (UID: "9cd22e64-3ad6-4f75-bc53-5dae3fccee49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:05.882710 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.882680    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:05.882842 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.882709    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:05.882842 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.882725    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:05.882842 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.882740    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:05.882842 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.882832    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-certs" pod "9cd22e64-3ad6-4f75-bc53-5dae3fccee49" (UID: "9cd22e64-3ad6-4f75-bc53-5dae3fccee49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:05.883921 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.883894    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-podinfo" pod "9cd22e64-3ad6-4f75-bc53-5dae3fccee49" (UID: "9cd22e64-3ad6-4f75-bc53-5dae3fccee49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:05.884063 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.884044    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-envoy" pod "9cd22e64-3ad6-4f75-bc53-5dae3fccee49" (UID: "9cd22e64-3ad6-4f75-bc53-5dae3fccee49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:05.884168 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.884145    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-kube-api-access-pflrq" pod "9cd22e64-3ad6-4f75-bc53-5dae3fccee49" (UID: "9cd22e64-3ad6-4f75-bc53-5dae3fccee49"). InnerVolumeSpecName "kube-api-access-pflrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:05.884334 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.884317    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-token" pod "9cd22e64-3ad6-4f75-bc53-5dae3fccee49" (UID: "9cd22e64-3ad6-4f75-bc53-5dae3fccee49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:05.984123 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.984048    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pflrq\" (UniqueName: \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-kube-api-access-pflrq\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:05.984123 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.984072    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:05.984123 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.984080    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:05.984123 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.984092    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:05.984123 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:05.984104    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cd22e64-3ad6-4f75-bc53-5dae3fccee49-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:06.634243 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:06.634203    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4" event={"ID":"9cd22e64-3ad6-4f75-bc53-5dae3fccee49","Type":"ContainerDied","Data":"1b62042a84463242f9f183d023c47534826732ab3cbcad02ad2e1747db9befa5"}
Oct 30 20:00:06.634697 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:06.634315    4420 scope.go:122] "RemoveContainer" containerID="faf7011b49cd8743c04e227ef5e87e95b96464589ccc4c12bf947fbca6c3474a"
Oct 30 20:00:06.634697 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:06.634492    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"
Oct 30 20:00:06.805180 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:06.805152    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"]
Oct 30 20:00:06.855754 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:06.855714    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-2e5977-openshift-default-57fc8dbbf-cjgv4"]
Oct 30 20:00:06.896484 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:06.896457    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" path="/var/lib/kubelet/pods/9cd22e64-3ad6-4f75-bc53-5dae3fccee49/volumes"
Oct 30 20:00:11.867286 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.867259    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"]
Oct 30 20:00:11.867794 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.867780    4420 cpu_manager.go:414] "containerMap: removing container" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy"
Oct 30 20:00:11.867794 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.867795    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy"
Oct 30 20:00:11.867880 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.867834    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a26e766b-7314-455c-ae4f-0d7acca29891" containerName="revoke-keys"
Oct 30 20:00:11.867880 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.867839    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26e766b-7314-455c-ae4f-0d7acca29891" containerName="revoke-keys"
Oct 30 20:00:11.867880 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.867852    4420 cpu_manager.go:414] "containerMap: removing container" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy"
Oct 30 20:00:11.867880 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.867858    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74aa3ef-91cc-4ece-b74e-c3332572e110" containerName="istio-proxy"
Oct 30 20:00:11.868002 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.867931    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy"
Oct 30 20:00:11.868002 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.867942    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26e766b-7314-455c-ae4f-0d7acca29891" containerName="revoke-keys"
Oct 30 20:00:11.868002 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.867950    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy"
Oct 30 20:00:11.872569 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.872552    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:11.937979 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.937956    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f8a9d8-openshift-default-dockercfg-dtxgv\""
Oct 30 20:00:11.948592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.948566    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-credential-socket\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:11.948698 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.948619    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-data\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:11.948745 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.948698    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istiod-ca-cert\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:11.948745 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.948731    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-token\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:11.948843 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.948762    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-socket\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:11.948910 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.948898    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m87l5\" (UniqueName: \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-kube-api-access-m87l5\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:11.948961 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.948950    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-podinfo\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:11.948993 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.948971    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-certs\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:11.949068 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.949040    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-envoy\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:11.982976 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:11.982951    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"]
Oct 30 20:00:12.052216 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.052173    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-token\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.052500 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.052477    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-socket\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.052684 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.052670    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m87l5\" (UniqueName: \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-kube-api-access-m87l5\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.052856 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.052842    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-podinfo\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.052966 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.052950    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-certs\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.053184 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.053159    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-envoy\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.053860 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.053838    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-credential-socket\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.054097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.054072    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-data\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.054299 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.054280    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istiod-ca-cert\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.055729 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.055706    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-socket\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.056532 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.056511    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istiod-ca-cert\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.068248 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.061106    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-credential-socket\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.068418 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.061475    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-data\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.068521 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.061504    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-podinfo\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.068610 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.061688    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-certs\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.070533 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.070469    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-envoy\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.112647 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.112621    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-token\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.112744 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.112724    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m87l5\" (UniqueName: \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-kube-api-access-m87l5\") pod \"e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") " pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.187600 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.187537    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:12.546789 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.546693    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" containerID="cri-o://a2691a4a171acfdf4ba92336f0e0dcdefeb951fb2a95b1879c5369d9ca420fc3" gracePeriod=45
Oct 30 20:00:12.631068 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.631041    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"]
Oct 30 20:00:12.632054 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:00:12.632025    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21239ee1_9c9e_44e2_ab73_4ab81c3def78.slice/crio-120e292fa4593ee1caa2386706c8d626814f9202b6453b863d0ed128c07e53d9 WatchSource:0}: Error finding container 120e292fa4593ee1caa2386706c8d626814f9202b6453b863d0ed128c07e53d9: Status 404 returned error can't find the container with id 120e292fa4593ee1caa2386706c8d626814f9202b6453b863d0ed128c07e53d9
Oct 30 20:00:12.652512 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.652479    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:00:12.652596 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.652549    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:00:12.652596 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.652580    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:00:12.672061 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.672033    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" event={"ID":"21239ee1-9c9e-44e2-ab73-4ab81c3def78","Type":"ContainerStarted","Data":"120e292fa4593ee1caa2386706c8d626814f9202b6453b863d0ed128c07e53d9"}
Oct 30 20:00:12.674108 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.674085    4420 generic.go:356] "Generic (PLEG): container finished" podID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerID="a2691a4a171acfdf4ba92336f0e0dcdefeb951fb2a95b1879c5369d9ca420fc3" exitCode=0
Oct 30 20:00:12.674244 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.674154    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" event={"ID":"db68fab2-eabb-414a-b123-9d5c6f61ee72","Type":"ContainerDied","Data":"a2691a4a171acfdf4ba92336f0e0dcdefeb951fb2a95b1879c5369d9ca420fc3"}
Oct 30 20:00:12.707359 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.707107    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused"
Oct 30 20:00:12.844026 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.844001    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 20:00:12.966803 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.966767    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-model-cache\") pod \"db68fab2-eabb-414a-b123-9d5c6f61ee72\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") "
Oct 30 20:00:12.975378 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.971943    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-model-cache" pod "db68fab2-eabb-414a-b123-9d5c6f61ee72" (UID: "db68fab2-eabb-414a-b123-9d5c6f61ee72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.975649 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.975628    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db68fab2-eabb-414a-b123-9d5c6f61ee72-kube-api-access-jqcjs\" (UniqueName: \"kubernetes.io/projected/db68fab2-eabb-414a-b123-9d5c6f61ee72-kube-api-access-jqcjs\") pod \"db68fab2-eabb-414a-b123-9d5c6f61ee72\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") "
Oct 30 20:00:12.976320 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.976299    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-home\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-home\") pod \"db68fab2-eabb-414a-b123-9d5c6f61ee72\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") "
Oct 30 20:00:12.976839 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.976671    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-home" pod "db68fab2-eabb-414a-b123-9d5c6f61ee72" (UID: "db68fab2-eabb-414a-b123-9d5c6f61ee72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.977103 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.976580    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-dshm\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-dshm\") pod \"db68fab2-eabb-414a-b123-9d5c6f61ee72\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") "
Oct 30 20:00:12.977433 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.977416    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-tmp-dir\") pod \"db68fab2-eabb-414a-b123-9d5c6f61ee72\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") "
Oct 30 20:00:12.977589 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.977575    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db68fab2-eabb-414a-b123-9d5c6f61ee72-tls-certs\" (UniqueName: \"kubernetes.io/secret/db68fab2-eabb-414a-b123-9d5c6f61ee72-tls-certs\") pod \"db68fab2-eabb-414a-b123-9d5c6f61ee72\" (UID: \"db68fab2-eabb-414a-b123-9d5c6f61ee72\") "
Oct 30 20:00:12.978752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.978733    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.980306 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.980290    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:12.980444 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.977725    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-tmp-dir" pod "db68fab2-eabb-414a-b123-9d5c6f61ee72" (UID: "db68fab2-eabb-414a-b123-9d5c6f61ee72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.980527 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.979888    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db68fab2-eabb-414a-b123-9d5c6f61ee72-kube-api-access-jqcjs" pod "db68fab2-eabb-414a-b123-9d5c6f61ee72" (UID: "db68fab2-eabb-414a-b123-9d5c6f61ee72"). InnerVolumeSpecName "kube-api-access-jqcjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:12.980611 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.979983    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-dshm" pod "db68fab2-eabb-414a-b123-9d5c6f61ee72" (UID: "db68fab2-eabb-414a-b123-9d5c6f61ee72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:12.983990 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:12.983808    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db68fab2-eabb-414a-b123-9d5c6f61ee72-tls-certs" pod "db68fab2-eabb-414a-b123-9d5c6f61ee72" (UID: "db68fab2-eabb-414a-b123-9d5c6f61ee72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:13.091049 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:13.090881    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqcjs\" (UniqueName: \"kubernetes.io/projected/db68fab2-eabb-414a-b123-9d5c6f61ee72-kube-api-access-jqcjs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:13.091049 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:13.090916    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:13.091049 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:13.090933    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db68fab2-eabb-414a-b123-9d5c6f61ee72-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:13.091049 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:13.090948    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db68fab2-eabb-414a-b123-9d5c6f61ee72-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:13.682419 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:13.682382    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" event={"ID":"21239ee1-9c9e-44e2-ab73-4ab81c3def78","Type":"ContainerStarted","Data":"6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951"}
Oct 30 20:00:13.684014 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:13.683992    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"
Oct 30 20:00:13.684130 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:13.683991    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2" event={"ID":"db68fab2-eabb-414a-b123-9d5c6f61ee72","Type":"ContainerDied","Data":"b8ac54ccd18db170d1305c9d603aa166311f3f0f1be64b32c3846c5b4905876c"}
Oct 30 20:00:13.684130 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:13.684121    4420 scope.go:122] "RemoveContainer" containerID="a2691a4a171acfdf4ba92336f0e0dcdefeb951fb2a95b1879c5369d9ca420fc3"
Oct 30 20:00:13.899368 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:13.899309    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" podStartSLOduration=2.899297335 podStartE2EDuration="2.899297335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:13.850046366 +0000 UTC m=+2661.061263511" watchObservedRunningTime="2026-10-30 20:00:13.899297335 +0000 UTC m=+2661.110514479"
Oct 30 20:00:13.962055 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:13.961985    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"]
Oct 30 20:00:13.966732 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:13.966709    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5f55cd/distinct-2-w6-5f55cd-kserve-8d545df5d-r59p2"]
Oct 30 20:00:14.188301 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:14.188272    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:14.194526 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:14.194503    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:14.691597 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:14.691567    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:14.692484 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:14.692461    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:00:14.896278 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:14.896248    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" path="/var/lib/kubelet/pods/db68fab2-eabb-414a-b123-9d5c6f61ee72/volumes"
Oct 30 20:00:19.672783 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.672747    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main" containerID="cri-o://b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0" gracePeriod=45
Oct 30 20:00:19.923592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.923539    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 20:00:19.965792 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.965762    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tls-certs\") pod \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") "
Oct 30 20:00:19.965953 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.965874    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tmp-dir\") pod \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") "
Oct 30 20:00:19.965953 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.965947    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-model-cache\") pod \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") "
Oct 30 20:00:19.966066 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.966041    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-home\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-home\") pod \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") "
Oct 30 20:00:19.966142 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.966124    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-dshm\") pod \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") "
Oct 30 20:00:19.966230 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.966195    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91701fe-f82f-4367-84c5-5a5ff7355fc1-kube-api-access-rwmqq\" (UniqueName: \"kubernetes.io/projected/f91701fe-f82f-4367-84c5-5a5ff7355fc1-kube-api-access-rwmqq\") pod \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\" (UID: \"f91701fe-f82f-4367-84c5-5a5ff7355fc1\") "
Oct 30 20:00:19.966282 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.966116    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tmp-dir" pod "f91701fe-f82f-4367-84c5-5a5ff7355fc1" (UID: "f91701fe-f82f-4367-84c5-5a5ff7355fc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:19.966401 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.966375    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-model-cache" pod "f91701fe-f82f-4367-84c5-5a5ff7355fc1" (UID: "f91701fe-f82f-4367-84c5-5a5ff7355fc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:19.966401 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.966385    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-home" pod "f91701fe-f82f-4367-84c5-5a5ff7355fc1" (UID: "f91701fe-f82f-4367-84c5-5a5ff7355fc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:19.966681 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.966661    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.967005 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.966688    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.967005 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.966702    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:19.968337 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.968321    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-dshm" pod "f91701fe-f82f-4367-84c5-5a5ff7355fc1" (UID: "f91701fe-f82f-4367-84c5-5a5ff7355fc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:19.968408 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.968389    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91701fe-f82f-4367-84c5-5a5ff7355fc1-kube-api-access-rwmqq" pod "f91701fe-f82f-4367-84c5-5a5ff7355fc1" (UID: "f91701fe-f82f-4367-84c5-5a5ff7355fc1"). InnerVolumeSpecName "kube-api-access-rwmqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:19.968485 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:19.968464    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tls-certs" pod "f91701fe-f82f-4367-84c5-5a5ff7355fc1" (UID: "f91701fe-f82f-4367-84c5-5a5ff7355fc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:20.068547 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.068518    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91701fe-f82f-4367-84c5-5a5ff7355fc1-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.068547 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.068549    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwmqq\" (UniqueName: \"kubernetes.io/projected/f91701fe-f82f-4367-84c5-5a5ff7355fc1-kube-api-access-rwmqq\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.068727 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.068562    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91701fe-f82f-4367-84c5-5a5ff7355fc1-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:20.726541 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.726508    4420 generic.go:356] "Generic (PLEG): container finished" podID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerID="b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0" exitCode=0
Oct 30 20:00:20.726989 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.726562    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"
Oct 30 20:00:20.726989 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.726571    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" event={"ID":"f91701fe-f82f-4367-84c5-5a5ff7355fc1","Type":"ContainerDied","Data":"b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0"}
Oct 30 20:00:20.726989 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.726601    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm" event={"ID":"f91701fe-f82f-4367-84c5-5a5ff7355fc1","Type":"ContainerDied","Data":"c03696583b260538c1d0d724214b7a9c51cabb75ad82aa59f80e1b3963f764c3"}
Oct 30 20:00:20.726989 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.726618    4420 scope.go:122] "RemoveContainer" containerID="b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0"
Oct 30 20:00:20.736783 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.736764    4420 scope.go:122] "RemoveContainer" containerID="b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0"
Oct 30 20:00:20.737109 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:00:20.737090    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0\": container with ID starting with b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0 not found: ID does not exist" containerID="b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0"
Oct 30 20:00:20.737183 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.737118    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0"} err="failed to get container status \"b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0\": rpc error: code = NotFound desc = could not find container \"b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0\": container with ID starting with b3006852f10efb1c245b68be203ba29b07bdf49959009a2f54bc050fc6fe2dc0 not found: ID does not exist"
Oct 30 20:00:20.896891 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.896866    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"]
Oct 30 20:00:20.940674 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:20.940647    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-686689/distinct-w5-686689-kserve-594c55c4c7-8zvlm"]
Oct 30 20:00:22.895342 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:22.895309    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" path="/var/lib/kubelet/pods/f91701fe-f82f-4367-84c5-5a5ff7355fc1/volumes"
Oct 30 20:00:24.262776 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.262742    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"]
Oct 30 20:00:24.263225 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.262944    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g" podUID="7b5b0a45-21fd-4669-b6b1-d7dcb94edb09" containerName="maas-api" containerID="cri-o://6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330" gracePeriod=30
Oct 30 20:00:24.534304 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.534276    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 20:00:24.618779 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.618755    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-maas-api-tls\") pod \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") "
Oct 30 20:00:24.618936 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.618845    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-metrics-certs\") pod \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") "
Oct 30 20:00:24.618936 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.618876    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-openshift-service-ca\") pod \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") "
Oct 30 20:00:24.618936 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.618909    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-kube-api-access-62kqd\" (UniqueName: \"kubernetes.io/projected/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-kube-api-access-62kqd\") pod \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\" (UID: \"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09\") "
Oct 30 20:00:24.619274 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.619253    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-openshift-service-ca" pod "7b5b0a45-21fd-4669-b6b1-d7dcb94edb09" (UID: "7b5b0a45-21fd-4669-b6b1-d7dcb94edb09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:24.621091 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.621069    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-kube-api-access-62kqd" pod "7b5b0a45-21fd-4669-b6b1-d7dcb94edb09" (UID: "7b5b0a45-21fd-4669-b6b1-d7dcb94edb09"). InnerVolumeSpecName "kube-api-access-62kqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:24.621091 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.621080    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-maas-api-tls" pod "7b5b0a45-21fd-4669-b6b1-d7dcb94edb09" (UID: "7b5b0a45-21fd-4669-b6b1-d7dcb94edb09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:24.621196 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.621101    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-metrics-certs" pod "7b5b0a45-21fd-4669-b6b1-d7dcb94edb09" (UID: "7b5b0a45-21fd-4669-b6b1-d7dcb94edb09"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:24.720688 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.720650    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:24.720688 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.720686    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:24.720688 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.720697    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:24.720929 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.720705    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62kqd\" (UniqueName: \"kubernetes.io/projected/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09-kube-api-access-62kqd\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:24.750270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.750245    4420 generic.go:356] "Generic (PLEG): container finished" podID="7b5b0a45-21fd-4669-b6b1-d7dcb94edb09" containerID="6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330" exitCode=0
Oct 30 20:00:24.750370 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.750294    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"
Oct 30 20:00:24.750370 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.750329    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g" event={"ID":"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09","Type":"ContainerDied","Data":"6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330"}
Oct 30 20:00:24.750452 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.750385    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g" event={"ID":"7b5b0a45-21fd-4669-b6b1-d7dcb94edb09","Type":"ContainerDied","Data":"8451bf0fa23150bbbc4129d78e0020726e28e8baa1777af291c192f80a573f5c"}
Oct 30 20:00:24.750452 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.750409    4420 scope.go:122] "RemoveContainer" containerID="6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330"
Oct 30 20:00:24.760224 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.760211    4420 scope.go:122] "RemoveContainer" containerID="6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330"
Oct 30 20:00:24.760487 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:00:24.760462    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330\": container with ID starting with 6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330 not found: ID does not exist" containerID="6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330"
Oct 30 20:00:24.760570 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.760496    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330"} err="failed to get container status \"6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330\": rpc error: code = NotFound desc = could not find container \"6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330\": container with ID starting with 6865397fd3c2f160185604f5dda35bde22f846a35d9f7f49403ef976b0842330 not found: ID does not exist"
Oct 30 20:00:24.918745 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.918707    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"]
Oct 30 20:00:24.921660 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:24.921638    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c5537-b44c897d7-glj5g"]
Oct 30 20:00:26.895096 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:26.895066    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b5b0a45-21fd-4669-b6b1-d7dcb94edb09" path="/var/lib/kubelet/pods/7b5b0a45-21fd-4669-b6b1-d7dcb94edb09/volumes"
Oct 30 20:00:34.204984 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:34.204953    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"]
Oct 30 20:00:34.205528 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:34.205157    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy" containerID="cri-o://2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd" gracePeriod=30
Oct 30 20:00:35.196326 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:00:35.196290    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:00:39.476191 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.476124    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 20:00:39.578469 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578442    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-data\") pod \"773cf534-eeb3-43e2-8616-264970478e1d\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") "
Oct 30 20:00:39.578596 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578480    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/773cf534-eeb3-43e2-8616-264970478e1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773cf534-eeb3-43e2-8616-264970478e1d-istiod-ca-cert\") pod \"773cf534-eeb3-43e2-8616-264970478e1d\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") "
Oct 30 20:00:39.578596 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578515    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-socket\") pod \"773cf534-eeb3-43e2-8616-264970478e1d\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") "
Oct 30 20:00:39.578596 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578546    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-istio-token\" (UniqueName: \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-istio-token\") pod \"773cf534-eeb3-43e2-8616-264970478e1d\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") "
Oct 30 20:00:39.578596 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578570    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/773cf534-eeb3-43e2-8616-264970478e1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773cf534-eeb3-43e2-8616-264970478e1d-istio-podinfo\") pod \"773cf534-eeb3-43e2-8616-264970478e1d\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") "
Oct 30 20:00:39.578806 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578638    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-certs\") pod \"773cf534-eeb3-43e2-8616-264970478e1d\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") "
Oct 30 20:00:39.578806 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578685    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-data" pod "773cf534-eeb3-43e2-8616-264970478e1d" (UID: "773cf534-eeb3-43e2-8616-264970478e1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:39.578806 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578721    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-kube-api-access-7bjsn\" (UniqueName: \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-kube-api-access-7bjsn\") pod \"773cf534-eeb3-43e2-8616-264970478e1d\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") "
Oct 30 20:00:39.578806 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578753    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-envoy\") pod \"773cf534-eeb3-43e2-8616-264970478e1d\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") "
Oct 30 20:00:39.579061 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578868    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-credential-socket\") pod \"773cf534-eeb3-43e2-8616-264970478e1d\" (UID: \"773cf534-eeb3-43e2-8616-264970478e1d\") "
Oct 30 20:00:39.579061 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578891    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-socket" pod "773cf534-eeb3-43e2-8616-264970478e1d" (UID: "773cf534-eeb3-43e2-8616-264970478e1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:39.579061 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.578968    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-certs" pod "773cf534-eeb3-43e2-8616-264970478e1d" (UID: "773cf534-eeb3-43e2-8616-264970478e1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:39.579061 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.579004    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773cf534-eeb3-43e2-8616-264970478e1d-istiod-ca-cert" pod "773cf534-eeb3-43e2-8616-264970478e1d" (UID: "773cf534-eeb3-43e2-8616-264970478e1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:39.579286 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.579260    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-credential-socket" pod "773cf534-eeb3-43e2-8616-264970478e1d" (UID: "773cf534-eeb3-43e2-8616-264970478e1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:39.579367 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.579344    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:39.579367 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.579359    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773cf534-eeb3-43e2-8616-264970478e1d-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:39.579471 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.579368    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:39.579471 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.579378    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:39.580975 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.580953    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-kube-api-access-7bjsn" pod "773cf534-eeb3-43e2-8616-264970478e1d" (UID: "773cf534-eeb3-43e2-8616-264970478e1d"). InnerVolumeSpecName "kube-api-access-7bjsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:39.580975 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.580962    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/773cf534-eeb3-43e2-8616-264970478e1d-istio-podinfo" pod "773cf534-eeb3-43e2-8616-264970478e1d" (UID: "773cf534-eeb3-43e2-8616-264970478e1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:00:39.581338 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.581320    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-istio-token" pod "773cf534-eeb3-43e2-8616-264970478e1d" (UID: "773cf534-eeb3-43e2-8616-264970478e1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:39.581428 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.581332    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-envoy" pod "773cf534-eeb3-43e2-8616-264970478e1d" (UID: "773cf534-eeb3-43e2-8616-264970478e1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:00:39.680322 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.680300    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:39.680322 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.680323    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773cf534-eeb3-43e2-8616-264970478e1d-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:39.680471 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.680332    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bjsn\" (UniqueName: \"kubernetes.io/projected/773cf534-eeb3-43e2-8616-264970478e1d-kube-api-access-7bjsn\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:39.680471 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.680340    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:39.680471 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.680349    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773cf534-eeb3-43e2-8616-264970478e1d-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:39.838081 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.838019    4420 generic.go:356] "Generic (PLEG): container finished" podID="773cf534-eeb3-43e2-8616-264970478e1d" containerID="2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd" exitCode=0
Oct 30 20:00:39.838081 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.838064    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" event={"ID":"773cf534-eeb3-43e2-8616-264970478e1d","Type":"ContainerDied","Data":"2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd"}
Oct 30 20:00:39.838242 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.838105    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l" event={"ID":"773cf534-eeb3-43e2-8616-264970478e1d","Type":"ContainerDied","Data":"a3a78acac45ba946306e50f36d22687e4a4463433f50b2e6a894d5cf1d106885"}
Oct 30 20:00:39.838242 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.838125    4420 scope.go:122] "RemoveContainer" containerID="2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd"
Oct 30 20:00:39.838242 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.838067    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"
Oct 30 20:00:39.848296 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.848279    4420 scope.go:122] "RemoveContainer" containerID="80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f"
Oct 30 20:00:39.857041 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.857023    4420 scope.go:122] "RemoveContainer" containerID="2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd"
Oct 30 20:00:39.857280 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:00:39.857264    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd\": container with ID starting with 2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd not found: ID does not exist" containerID="2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd"
Oct 30 20:00:39.857330 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.857288    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd"} err="failed to get container status \"2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd\": rpc error: code = NotFound desc = could not find container \"2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd\": container with ID starting with 2eb4370aaa928c46d05db6a30104f96b79e0be5f270440fc20e5040c2f68dabd not found: ID does not exist"
Oct 30 20:00:39.857330 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.857304    4420 scope.go:122] "RemoveContainer" containerID="80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f"
Oct 30 20:00:39.857526 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:00:39.857507    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f\": container with ID starting with 80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f not found: ID does not exist" containerID="80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f"
Oct 30 20:00:39.857564 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:39.857533    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f"} err="failed to get container status \"80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f\": rpc error: code = NotFound desc = could not find container \"80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f\": container with ID starting with 80283da3f617468fd6ad59044b75c85607a0cba4c0bcdc4dda6958fa6d57543f not found: ID does not exist"
Oct 30 20:00:40.003224 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:40.003200    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"]
Oct 30 20:00:40.008603 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:40.008580    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1c5537-openshift-default-5df844678c-z827l"]
Oct 30 20:00:40.895665 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:40.895636    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="773cf534-eeb3-43e2-8616-264970478e1d" path="/var/lib/kubelet/pods/773cf534-eeb3-43e2-8616-264970478e1d/volumes"
Oct 30 20:00:44.745056 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745022    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"]
Oct 30 20:00:44.745619 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745602    4420 cpu_manager.go:414] "containerMap: removing container" podUID="7b5b0a45-21fd-4669-b6b1-d7dcb94edb09" containerName="maas-api"
Oct 30 20:00:44.745663 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745622    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5b0a45-21fd-4669-b6b1-d7dcb94edb09" containerName="maas-api"
Oct 30 20:00:44.745663 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745636    4420 cpu_manager.go:414] "containerMap: removing container" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy"
Oct 30 20:00:44.745663 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745640    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd22e64-3ad6-4f75-bc53-5dae3fccee49" containerName="istio-proxy"
Oct 30 20:00:44.745663 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745648    4420 cpu_manager.go:414] "containerMap: removing container" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main"
Oct 30 20:00:44.745663 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745654    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main"
Oct 30 20:00:44.745663 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745663    4420 cpu_manager.go:414] "containerMap: removing container" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy"
Oct 30 20:00:44.745920 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745668    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy"
Oct 30 20:00:44.745920 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745699    4420 cpu_manager.go:414] "containerMap: removing container" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main"
Oct 30 20:00:44.745920 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745703    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main"
Oct 30 20:00:44.745920 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745771    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy"
Oct 30 20:00:44.745920 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745779    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91701fe-f82f-4367-84c5-5a5ff7355fc1" containerName="main"
Oct 30 20:00:44.745920 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745787    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5b0a45-21fd-4669-b6b1-d7dcb94edb09" containerName="maas-api"
Oct 30 20:00:44.745920 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745792    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="db68fab2-eabb-414a-b123-9d5c6f61ee72" containerName="main"
Oct 30 20:00:44.745920 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.745798    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy"
Oct 30 20:00:44.750290 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.750272    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:44.802653 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.802632    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5f55cd-openshift-default-dockercfg-wh68v\""
Oct 30 20:00:44.854518 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.854493    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"]
Oct 30 20:00:44.937890 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.937863    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:44.938015 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.937892    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn7bl\" (UniqueName: \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-kube-api-access-gn7bl\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:44.938015 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.937931    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:44.938091 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.938046    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:44.938125 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.938094    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:44.938199 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.938158    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:44.938257 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.938244    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:44.938291 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.938274    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:44.938326 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:44.938290    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039137 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039079    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039137 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039109    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn7bl\" (UniqueName: \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-kube-api-access-gn7bl\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039137 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039128    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039306 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039184    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039306 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039241    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039306 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039270    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039306 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039293    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039482 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039324    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039482 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039359    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039781 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039758    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-credential-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039899 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039669    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-socket\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039899 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039617    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-certs\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039899 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039870    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istiod-ca-cert\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.039899 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.039874    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-data\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.041451 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.041431    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-envoy\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.041873 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.041855    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-podinfo\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.101781 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.101757    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn7bl\" (UniqueName: \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-kube-api-access-gn7bl\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.101897 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.101809    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-token\") pod \"e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") " pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.364045 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.363967    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:45.803196 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.803169    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"]
Oct 30 20:00:45.803828 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:00:45.803795    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2702ec_d3bf_45f9_a6cd_cfff1b296d0e.slice/crio-8502cda12158d1ffcecdecaeff0ce0a057959da09f88618bb87e7a06bafaf19e WatchSource:0}: Error finding container 8502cda12158d1ffcecdecaeff0ce0a057959da09f88618bb87e7a06bafaf19e: Status 404 returned error can't find the container with id 8502cda12158d1ffcecdecaeff0ce0a057959da09f88618bb87e7a06bafaf19e
Oct 30 20:00:45.806163 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.806129    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:00:45.806246 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.806211    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:00:45.806246 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.806240    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:00:45.881080 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.881056    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" event={"ID":"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e","Type":"ContainerStarted","Data":"059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865"}
Oct 30 20:00:45.881187 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:45.881106    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" event={"ID":"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e","Type":"ContainerStarted","Data":"8502cda12158d1ffcecdecaeff0ce0a057959da09f88618bb87e7a06bafaf19e"}
Oct 30 20:00:46.012926 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.012874    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" podStartSLOduration=2.012860841 podStartE2EDuration="2.012860841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:46.0093146 +0000 UTC m=+2693.220531817" watchObservedRunningTime="2026-10-30 20:00:46.012860841 +0000 UTC m=+2693.224077984"
Oct 30 20:00:46.364315 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.364283    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:46.365461 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.365440    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:46.365995 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.365958    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 20:00:46.379073 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.379045    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"]
Oct 30 20:00:46.379369 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.379310    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr" podUID="d1f8018e-cf10-47a5-bf36-5e1333e199e0" containerName="maas-api" containerID="cri-o://4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253" gracePeriod=30
Oct 30 20:00:46.698642 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.698617    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 20:00:46.788840 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.788770    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1f8018e-cf10-47a5-bf36-5e1333e199e0-kube-api-access-4nnf6\" (UniqueName: \"kubernetes.io/projected/d1f8018e-cf10-47a5-bf36-5e1333e199e0-kube-api-access-4nnf6\") pod \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") "
Oct 30 20:00:46.789021 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.788895    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1f8018e-cf10-47a5-bf36-5e1333e199e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f8018e-cf10-47a5-bf36-5e1333e199e0-openshift-service-ca\") pod \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") "
Oct 30 20:00:46.789021 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.788929    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-metrics-certs\") pod \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") "
Oct 30 20:00:46.797887 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.793176    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f8018e-cf10-47a5-bf36-5e1333e199e0-openshift-service-ca" pod "d1f8018e-cf10-47a5-bf36-5e1333e199e0" (UID: "d1f8018e-cf10-47a5-bf36-5e1333e199e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:00:46.798772 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.798294    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-metrics-certs" pod "d1f8018e-cf10-47a5-bf36-5e1333e199e0" (UID: "d1f8018e-cf10-47a5-bf36-5e1333e199e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:46.803369 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.802465    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f8018e-cf10-47a5-bf36-5e1333e199e0-kube-api-access-4nnf6" pod "d1f8018e-cf10-47a5-bf36-5e1333e199e0" (UID: "d1f8018e-cf10-47a5-bf36-5e1333e199e0"). InnerVolumeSpecName "kube-api-access-4nnf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:00:46.889112 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.889077    4420 generic.go:356] "Generic (PLEG): container finished" podID="d1f8018e-cf10-47a5-bf36-5e1333e199e0" containerID="4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253" exitCode=0
Oct 30 20:00:46.889303 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.889186    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"
Oct 30 20:00:46.890011 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.889982    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-maas-api-tls\") pod \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\" (UID: \"d1f8018e-cf10-47a5-bf36-5e1333e199e0\") "
Oct 30 20:00:46.890483 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.890455    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nnf6\" (UniqueName: \"kubernetes.io/projected/d1f8018e-cf10-47a5-bf36-5e1333e199e0-kube-api-access-4nnf6\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:46.890483 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.890477    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f8018e-cf10-47a5-bf36-5e1333e199e0-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:46.890636 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.890488    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:46.892361 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.892342    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-maas-api-tls" pod "d1f8018e-cf10-47a5-bf36-5e1333e199e0" (UID: "d1f8018e-cf10-47a5-bf36-5e1333e199e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:00:46.895920 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.895898    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr" event={"ID":"d1f8018e-cf10-47a5-bf36-5e1333e199e0","Type":"ContainerDied","Data":"4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253"}
Oct 30 20:00:46.896013 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.895936    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr" event={"ID":"d1f8018e-cf10-47a5-bf36-5e1333e199e0","Type":"ContainerDied","Data":"945e998ffe6e828aabe22dfe5834a898bc72fd5aaef352b687533f24a3c3f000"}
Oct 30 20:00:46.896013 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.895967    4420 scope.go:122] "RemoveContainer" containerID="4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253"
Oct 30 20:00:46.905150 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.905133    4420 scope.go:122] "RemoveContainer" containerID="4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253"
Oct 30 20:00:46.905406 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:00:46.905386    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253\": container with ID starting with 4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253 not found: ID does not exist" containerID="4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253"
Oct 30 20:00:46.905443 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.905419    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253"} err="failed to get container status \"4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253\": rpc error: code = NotFound desc = could not find container \"4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253\": container with ID starting with 4f061a05655a9051ae4b94f606cb0c4a666a4cb2f29ca7266a70566e53ca9253 not found: ID does not exist"
Oct 30 20:00:46.991390 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:46.991341    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1f8018e-cf10-47a5-bf36-5e1333e199e0-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:00:47.314333 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:47.314297    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"]
Oct 30 20:00:47.319054 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:47.319029    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6ac304-76b84bf9c9-ppbwr"]
Oct 30 20:00:47.382884 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:47.380499    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:47.382884 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:47.380572    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 20:00:48.377841 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:48.376199    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:48.377841 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:48.376260    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 20:00:48.897225 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:48.897193    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1f8018e-cf10-47a5-bf36-5e1333e199e0" path="/var/lib/kubelet/pods/d1f8018e-cf10-47a5-bf36-5e1333e199e0/volumes"
Oct 30 20:00:49.372889 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:49.365076    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:49.372889 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:49.365154    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 20:00:50.374139 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:50.374105    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:50.382925 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:50.382883    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 20:00:51.377840 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:51.369106    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:51.377840 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:51.369166    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 20:00:52.365096 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:52.365065    4420 patch_prober.go:29] interesting pod/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 30 20:00:52.365266 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:52.365131    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 30 20:00:53.475903 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:53.475866    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:53.476467 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:53.476443    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:53.477759 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:53.477734    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:00:58.720144 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.720117    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"]
Oct 30 20:00:58.720611 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.720598    4420 cpu_manager.go:414] "containerMap: removing container" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy"
Oct 30 20:00:58.720653 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.720611    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773cf534-eeb3-43e2-8616-264970478e1d" containerName="istio-proxy"
Oct 30 20:00:58.720653 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.720636    4420 cpu_manager.go:414] "containerMap: removing container" podUID="d1f8018e-cf10-47a5-bf36-5e1333e199e0" containerName="maas-api"
Oct 30 20:00:58.720653 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.720641    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1f8018e-cf10-47a5-bf36-5e1333e199e0" containerName="maas-api"
Oct 30 20:00:58.720739 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.720728    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1f8018e-cf10-47a5-bf36-5e1333e199e0" containerName="maas-api"
Oct 30 20:00:58.725857 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.725841    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:58.827183 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.827154    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqqq8\" (UniqueName: \"kubernetes.io/projected/21ece487-7e77-4656-8f3c-ea9131e437bf-kube-api-access-wqqq8\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:58.827320 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.827212    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ece487-7e77-4656-8f3c-ea9131e437bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:58.827388 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.827342    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-metrics-certs\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:58.827388 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.827372    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:58.830294 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.830267    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f8a9d8-serving-cert\""
Oct 30 20:00:58.833396 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.833366    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"]
Oct 30 20:00:58.928145 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.928116    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqqq8\" (UniqueName: \"kubernetes.io/projected/21ece487-7e77-4656-8f3c-ea9131e437bf-kube-api-access-wqqq8\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:58.928295 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.928173    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ece487-7e77-4656-8f3c-ea9131e437bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:58.928295 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.928228    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-metrics-certs\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:58.928295 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.928247    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:58.928475 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:00:58.928412    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-serving-cert: secret "maas-api-e2e-inf-b-f8a9d8-serving-cert" not found
Oct 30 20:00:58.928475 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:00:58.928474    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-maas-api-tls podName:21ece487-7e77-4656-8f3c-ea9131e437bf nodeName:}" failed. No retries permitted until 2026-10-30 20:00:59.428458804 +0000 UTC m=+2706.639675930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-maas-api-tls") pod "maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc" (UID: "21ece487-7e77-4656-8f3c-ea9131e437bf") : secret "maas-api-e2e-inf-b-f8a9d8-serving-cert" not found
Oct 30 20:00:58.929005 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.928936    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ece487-7e77-4656-8f3c-ea9131e437bf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:58.930699 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.930677    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-metrics-certs\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:58.988279 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:58.988212    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqqq8\" (UniqueName: \"kubernetes.io/projected/21ece487-7e77-4656-8f3c-ea9131e437bf-kube-api-access-wqqq8\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:59.434486 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:59.434450    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:59.436860 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:59.436841    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-maas-api-tls\") pod \"maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:00:59.638601 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:00:59.638578    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:01:00.012264 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:00.012244    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"]
Oct 30 20:01:00.014203 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:01:00.014165    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ece487_7e77_4656_8f3c_ea9131e437bf.slice/crio-a90e66f8a7ed93bd8f741dff55d5b37ececf0535ba0bd18f8314d5d416250b75 WatchSource:0}: Error finding container a90e66f8a7ed93bd8f741dff55d5b37ececf0535ba0bd18f8314d5d416250b75: Status 404 returned error can't find the container with id a90e66f8a7ed93bd8f741dff55d5b37ececf0535ba0bd18f8314d5d416250b75
Oct 30 20:01:00.971122 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:00.971088    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc" event={"ID":"21ece487-7e77-4656-8f3c-ea9131e437bf","Type":"ContainerStarted","Data":"a90e66f8a7ed93bd8f741dff55d5b37ececf0535ba0bd18f8314d5d416250b75"}
Oct 30 20:01:01.827985 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:01.827955    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"]
Oct 30 20:01:01.978258 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:01.978224    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc" event={"ID":"21ece487-7e77-4656-8f3c-ea9131e437bf","Type":"ContainerStarted","Data":"8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b"}
Oct 30 20:01:01.978435 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:01.978290    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:01:02.099181 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:02.099097    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc" podStartSLOduration=3.210407659 podStartE2EDuration="4.099083165s" totalImagesPullingTime="888.675506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:02.096346849 +0000 UTC m=+2709.307563993" watchObservedRunningTime="2026-10-30 20:01:02.099083165 +0000 UTC m=+2709.310300309"
Oct 30 20:01:03.364506 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.364470    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"]
Oct 30 20:01:03.365006 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.364726    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd" podUID="123635db-8c61-493e-b18d-e17afa4203bc" containerName="maas-api" containerID="cri-o://c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3" gracePeriod=30
Oct 30 20:01:03.644863 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.644802    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 20:01:03.788423 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.788396    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-maas-api-tls\") pod \"123635db-8c61-493e-b18d-e17afa4203bc\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") "
Oct 30 20:01:03.788585 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.788462    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/123635db-8c61-493e-b18d-e17afa4203bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/123635db-8c61-493e-b18d-e17afa4203bc-openshift-service-ca\") pod \"123635db-8c61-493e-b18d-e17afa4203bc\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") "
Oct 30 20:01:03.788660 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.788643    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-metrics-certs\") pod \"123635db-8c61-493e-b18d-e17afa4203bc\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") "
Oct 30 20:01:03.788718 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.788706    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/123635db-8c61-493e-b18d-e17afa4203bc-kube-api-access-69c92\" (UniqueName: \"kubernetes.io/projected/123635db-8c61-493e-b18d-e17afa4203bc-kube-api-access-69c92\") pod \"123635db-8c61-493e-b18d-e17afa4203bc\" (UID: \"123635db-8c61-493e-b18d-e17afa4203bc\") "
Oct 30 20:01:03.788889 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.788868    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/123635db-8c61-493e-b18d-e17afa4203bc-openshift-service-ca" pod "123635db-8c61-493e-b18d-e17afa4203bc" (UID: "123635db-8c61-493e-b18d-e17afa4203bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:03.789196 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.789171    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/123635db-8c61-493e-b18d-e17afa4203bc-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:03.790835 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.790774    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-metrics-certs" pod "123635db-8c61-493e-b18d-e17afa4203bc" (UID: "123635db-8c61-493e-b18d-e17afa4203bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:03.790997 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.790974    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/123635db-8c61-493e-b18d-e17afa4203bc-kube-api-access-69c92" pod "123635db-8c61-493e-b18d-e17afa4203bc" (UID: "123635db-8c61-493e-b18d-e17afa4203bc"). InnerVolumeSpecName "kube-api-access-69c92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:03.791548 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.791524    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-maas-api-tls" pod "123635db-8c61-493e-b18d-e17afa4203bc" (UID: "123635db-8c61-493e-b18d-e17afa4203bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:03.889950 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.889930    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:03.889950 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.889951    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69c92\" (UniqueName: \"kubernetes.io/projected/123635db-8c61-493e-b18d-e17afa4203bc-kube-api-access-69c92\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:03.890068 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.889961    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/123635db-8c61-493e-b18d-e17afa4203bc-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:03.990603 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.990573    4420 generic.go:356] "Generic (PLEG): container finished" podID="123635db-8c61-493e-b18d-e17afa4203bc" containerID="c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3" exitCode=0
Oct 30 20:01:03.990791 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.990626    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"
Oct 30 20:01:03.990791 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.990667    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd" event={"ID":"123635db-8c61-493e-b18d-e17afa4203bc","Type":"ContainerDied","Data":"c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3"}
Oct 30 20:01:03.990791 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.990732    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd" event={"ID":"123635db-8c61-493e-b18d-e17afa4203bc","Type":"ContainerDied","Data":"55595b2aa79d3edafa291d7e153c2e8dc5d853d307598c565b03625561b66e55"}
Oct 30 20:01:03.990791 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:03.990762    4420 scope.go:122] "RemoveContainer" containerID="c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3"
Oct 30 20:01:04.000529 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:04.000515    4420 scope.go:122] "RemoveContainer" containerID="c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3"
Oct 30 20:01:04.000761 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:01:04.000745    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3\": container with ID starting with c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3 not found: ID does not exist" containerID="c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3"
Oct 30 20:01:04.000800 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:04.000772    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3"} err="failed to get container status \"c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3\": rpc error: code = NotFound desc = could not find container \"c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3\": container with ID starting with c546714896365bbae179f605f2a28a28f66a53b3b8f75e08ece190ccc36b59d3 not found: ID does not exist"
Oct 30 20:01:04.161105 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:04.161049    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"]
Oct 30 20:01:04.166368 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:04.166346    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cf9b6f-5cbb8c5c96-s5qxd"]
Oct 30 20:01:04.895368 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:04.895326    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="123635db-8c61-493e-b18d-e17afa4203bc" path="/var/lib/kubelet/pods/123635db-8c61-493e-b18d-e17afa4203bc/volumes"
Oct 30 20:01:05.198764 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:01:05.198664    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:01:07.988968 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:07.988940    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:01:16.847925 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:16.847878    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main" containerID="cri-o://9a2b97b50d737874e5045284244cd09cb1b53c0b420452f62e23c4ef2a14f074" gracePeriod=45
Oct 30 20:01:17.063693 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.063436    4420 generic.go:356] "Generic (PLEG): container finished" podID="c44abbb2-14b5-4553-b03d-e29000457db2" containerID="9a2b97b50d737874e5045284244cd09cb1b53c0b420452f62e23c4ef2a14f074" exitCode=0
Oct 30 20:01:17.063693 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.063644    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" event={"ID":"c44abbb2-14b5-4553-b03d-e29000457db2","Type":"ContainerDied","Data":"9a2b97b50d737874e5045284244cd09cb1b53c0b420452f62e23c4ef2a14f074"}
Oct 30 20:01:17.124650 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.124627    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 20:01:17.233011 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.232979    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-dshm\") pod \"c44abbb2-14b5-4553-b03d-e29000457db2\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") "
Oct 30 20:01:17.233188 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.233069    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44abbb2-14b5-4553-b03d-e29000457db2-kube-api-access-xslt2\" (UniqueName: \"kubernetes.io/projected/c44abbb2-14b5-4553-b03d-e29000457db2-kube-api-access-xslt2\") pod \"c44abbb2-14b5-4553-b03d-e29000457db2\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") "
Oct 30 20:01:17.233188 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.233147    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c44abbb2-14b5-4553-b03d-e29000457db2-tls-certs\" (UniqueName: \"kubernetes.io/secret/c44abbb2-14b5-4553-b03d-e29000457db2-tls-certs\") pod \"c44abbb2-14b5-4553-b03d-e29000457db2\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") "
Oct 30 20:01:17.233336 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.233213    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-tmp-dir\") pod \"c44abbb2-14b5-4553-b03d-e29000457db2\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") "
Oct 30 20:01:17.233336 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.233259    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-model-cache\") pod \"c44abbb2-14b5-4553-b03d-e29000457db2\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") "
Oct 30 20:01:17.233448 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.233363    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-home\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-home\") pod \"c44abbb2-14b5-4553-b03d-e29000457db2\" (UID: \"c44abbb2-14b5-4553-b03d-e29000457db2\") "
Oct 30 20:01:17.234040 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.234016    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-home" pod "c44abbb2-14b5-4553-b03d-e29000457db2" (UID: "c44abbb2-14b5-4553-b03d-e29000457db2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.234648 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.234628    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-tmp-dir" pod "c44abbb2-14b5-4553-b03d-e29000457db2" (UID: "c44abbb2-14b5-4553-b03d-e29000457db2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.234897 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.234872    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-model-cache" pod "c44abbb2-14b5-4553-b03d-e29000457db2" (UID: "c44abbb2-14b5-4553-b03d-e29000457db2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.236099 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.236067    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-dshm" pod "c44abbb2-14b5-4553-b03d-e29000457db2" (UID: "c44abbb2-14b5-4553-b03d-e29000457db2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:17.236858 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.236796    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44abbb2-14b5-4553-b03d-e29000457db2-tls-certs" pod "c44abbb2-14b5-4553-b03d-e29000457db2" (UID: "c44abbb2-14b5-4553-b03d-e29000457db2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:17.244387 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.244032    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44abbb2-14b5-4553-b03d-e29000457db2-kube-api-access-xslt2" pod "c44abbb2-14b5-4553-b03d-e29000457db2" (UID: "c44abbb2-14b5-4553-b03d-e29000457db2"). InnerVolumeSpecName "kube-api-access-xslt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:17.335739 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.335702    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.335739 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.335741    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xslt2\" (UniqueName: \"kubernetes.io/projected/c44abbb2-14b5-4553-b03d-e29000457db2-kube-api-access-xslt2\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.336040 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.335757    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44abbb2-14b5-4553-b03d-e29000457db2-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.336040 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.335794    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.336040 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.335807    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:17.336040 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:17.335838    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44abbb2-14b5-4553-b03d-e29000457db2-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:18.087135 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:18.087091    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf" event={"ID":"c44abbb2-14b5-4553-b03d-e29000457db2","Type":"ContainerDied","Data":"b99794922e0f9ba66ea79bbc42fe2ff28d3de1b5c03e5df1badb8309d37842e5"}
Oct 30 20:01:18.087598 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:18.087141    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"
Oct 30 20:01:18.087598 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:18.087153    4420 scope.go:122] "RemoveContainer" containerID="9a2b97b50d737874e5045284244cd09cb1b53c0b420452f62e23c4ef2a14f074"
Oct 30 20:01:18.257292 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:18.256880    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"]
Oct 30 20:01:18.305281 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:18.305247    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2d9861/simulator-w0-2d9861-kserve-64dd6f6db9-qvmdf"]
Oct 30 20:01:18.896102 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:18.896072    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" path="/var/lib/kubelet/pods/c44abbb2-14b5-4553-b03d-e29000457db2/volumes"
Oct 30 20:01:25.468950 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.468912    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"]
Oct 30 20:01:25.469425 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.469201    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g" podUID="dab3dae4-c6eb-4256-95a1-c5354500fba4" containerName="maas-api" containerID="cri-o://9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48" gracePeriod=30
Oct 30 20:01:25.739264 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.739241    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 20:01:25.861579 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.858409    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab3dae4-c6eb-4256-95a1-c5354500fba4-kube-api-access-6njht\" (UniqueName: \"kubernetes.io/projected/dab3dae4-c6eb-4256-95a1-c5354500fba4-kube-api-access-6njht\") pod \"dab3dae4-c6eb-4256-95a1-c5354500fba4\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") "
Oct 30 20:01:25.861579 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.858513    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-maas-api-tls\") pod \"dab3dae4-c6eb-4256-95a1-c5354500fba4\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") "
Oct 30 20:01:25.861579 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.858616    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-metrics-certs\") pod \"dab3dae4-c6eb-4256-95a1-c5354500fba4\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") "
Oct 30 20:01:25.861579 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.858689    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dab3dae4-c6eb-4256-95a1-c5354500fba4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab3dae4-c6eb-4256-95a1-c5354500fba4-openshift-service-ca\") pod \"dab3dae4-c6eb-4256-95a1-c5354500fba4\" (UID: \"dab3dae4-c6eb-4256-95a1-c5354500fba4\") "
Oct 30 20:01:25.861579 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.859883    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab3dae4-c6eb-4256-95a1-c5354500fba4-openshift-service-ca" pod "dab3dae4-c6eb-4256-95a1-c5354500fba4" (UID: "dab3dae4-c6eb-4256-95a1-c5354500fba4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:25.863210 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.863182    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-maas-api-tls" pod "dab3dae4-c6eb-4256-95a1-c5354500fba4" (UID: "dab3dae4-c6eb-4256-95a1-c5354500fba4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:25.863422 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.863397    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab3dae4-c6eb-4256-95a1-c5354500fba4-kube-api-access-6njht" pod "dab3dae4-c6eb-4256-95a1-c5354500fba4" (UID: "dab3dae4-c6eb-4256-95a1-c5354500fba4"). InnerVolumeSpecName "kube-api-access-6njht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:25.864293 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.864267    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-metrics-certs" pod "dab3dae4-c6eb-4256-95a1-c5354500fba4" (UID: "dab3dae4-c6eb-4256-95a1-c5354500fba4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:25.960564 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.960384    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6njht\" (UniqueName: \"kubernetes.io/projected/dab3dae4-c6eb-4256-95a1-c5354500fba4-kube-api-access-6njht\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:25.960564 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.960419    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:25.960564 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.960433    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dab3dae4-c6eb-4256-95a1-c5354500fba4-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:25.960564 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:25.960449    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dab3dae4-c6eb-4256-95a1-c5354500fba4-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:26.137788 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:26.137752    4420 generic.go:356] "Generic (PLEG): container finished" podID="dab3dae4-c6eb-4256-95a1-c5354500fba4" containerID="9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48" exitCode=0
Oct 30 20:01:26.140967 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:26.137792    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"
Oct 30 20:01:26.140967 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:26.137859    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g" event={"ID":"dab3dae4-c6eb-4256-95a1-c5354500fba4","Type":"ContainerDied","Data":"9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48"}
Oct 30 20:01:26.140967 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:26.137934    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g" event={"ID":"dab3dae4-c6eb-4256-95a1-c5354500fba4","Type":"ContainerDied","Data":"501116e5a720ede9d9e62a4b352823dd53e9353e035284ddd2fc545d22ad22e6"}
Oct 30 20:01:26.140967 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:26.137967    4420 scope.go:122] "RemoveContainer" containerID="9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48"
Oct 30 20:01:26.151753 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:26.151732    4420 scope.go:122] "RemoveContainer" containerID="9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48"
Oct 30 20:01:26.152074 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:01:26.152050    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48\": container with ID starting with 9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48 not found: ID does not exist" containerID="9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48"
Oct 30 20:01:26.152156 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:26.152088    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48"} err="failed to get container status \"9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48\": rpc error: code = NotFound desc = could not find container \"9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48\": container with ID starting with 9a44bde0683f1654e581e28553848868c10adf21d1248e5e752ca038dafe0f48 not found: ID does not exist"
Oct 30 20:01:26.303230 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:26.303202    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"]
Oct 30 20:01:26.307264 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:26.307240    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-686689-847f5c9b48-gtg6g"]
Oct 30 20:01:26.896773 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:26.896748    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab3dae4-c6eb-4256-95a1-c5354500fba4" path="/var/lib/kubelet/pods/dab3dae4-c6eb-4256-95a1-c5354500fba4/volumes"
Oct 30 20:01:27.154476 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:27.154399    4420 generic.go:356] "Generic (PLEG): container finished" podID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerID="6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951" exitCode=0
Oct 30 20:01:27.154646 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:27.154478    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" event={"ID":"21239ee1-9c9e-44e2-ab73-4ab81c3def78","Type":"ContainerDied","Data":"6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951"}
Oct 30 20:01:27.154937 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:27.154920    4420 scope.go:122] "RemoveContainer" containerID="6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951"
Oct 30 20:01:27.156232 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:27.156205    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:01:27.156292 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:27.156272    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:01:27.156337 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:27.156302    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:01:27.561263 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:27.561140    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"]
Oct 30 20:01:27.564650 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:27.564622    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-686689-cmlwd"]
Oct 30 20:01:28.184954 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:28.184916    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" event={"ID":"21239ee1-9c9e-44e2-ab73-4ab81c3def78","Type":"ContainerStarted","Data":"3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957"}
Oct 30 20:01:28.188474 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:28.188454    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:01:28.189698 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:28.189669    4420 patch_prober.go:29] interesting pod/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 20:01:28.189841 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:28.189723    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 20:01:28.895642 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:28.895607    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26e766b-7314-455c-ae4f-0d7acca29891" path="/var/lib/kubelet/pods/a26e766b-7314-455c-ae4f-0d7acca29891/volumes"
Oct 30 20:01:29.189189 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:29.189103    4420 patch_prober.go:29] interesting pod/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 20:01:29.189189 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:29.189159    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 20:01:29.297412 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:29.297368    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"]
Oct 30 20:01:29.298209 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:29.298148    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerName="istio-proxy" containerID="cri-o://e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f" gracePeriod=30
Oct 30 20:01:29.304010 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:29.303986    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"]
Oct 30 20:01:29.304373 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:29.304338    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerName="istio-proxy" containerID="cri-o://317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8" gracePeriod=30
Oct 30 20:01:29.692347 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:29.692318    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:01:30.188678 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:30.188637    4420 patch_prober.go:29] interesting pod/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 20:01:30.188930 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:30.188698    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 20:01:31.188237 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:31.188202    4420 patch_prober.go:29] interesting pod/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 20:01:31.188720 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:31.188691    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 20:01:31.483691 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:31.483595    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:01:31.483691 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:31.483656    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:01:32.188211 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:32.188172    4420 patch_prober.go:29] interesting pod/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 30 20:01:32.188397 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:32.188237    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 30 20:01:32.350475 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:32.350443    4420 patch_prober.go:29] interesting pod/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:01:32.350651 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:32.350493    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:01:33.261443 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:33.261409    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:01:33.264586 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:33.264564    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:01:34.512865 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.512831    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"]
Oct 30 20:01:34.513433 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.513092    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" podUID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" containerName="istio-proxy" containerID="cri-o://a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab" gracePeriod=30
Oct 30 20:01:34.623256 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.623228    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 20:01:34.627192 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.627171    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 20:01:34.705019 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.704982    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-envoy\") pod \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") "
Oct 30 20:01:34.705169 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705064    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-kube-api-access-2thqk\" (UniqueName: \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-kube-api-access-2thqk\") pod \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") "
Oct 30 20:01:34.705169 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705161    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-credential-socket\") pod \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") "
Oct 30 20:01:34.705296 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705184    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-podinfo\") pod \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") "
Oct 30 20:01:34.705296 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705214    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-socket\") pod \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") "
Oct 30 20:01:34.705296 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705277    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istiod-ca-cert\") pod \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") "
Oct 30 20:01:34.705447 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705304    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-token\" (UniqueName: \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-token\") pod \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") "
Oct 30 20:01:34.705447 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705347    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-data\") pod \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") "
Oct 30 20:01:34.705447 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705386    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-certs\") pod \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\" (UID: \"b12b03fa-7887-40d9-bcc6-8c71f815ad0b\") "
Oct 30 20:01:34.705603 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705484    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-credential-socket" pod "b12b03fa-7887-40d9-bcc6-8c71f815ad0b" (UID: "b12b03fa-7887-40d9-bcc6-8c71f815ad0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.705603 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705492    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-socket" pod "b12b03fa-7887-40d9-bcc6-8c71f815ad0b" (UID: "b12b03fa-7887-40d9-bcc6-8c71f815ad0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.705705 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705683    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-data" pod "b12b03fa-7887-40d9-bcc6-8c71f815ad0b" (UID: "b12b03fa-7887-40d9-bcc6-8c71f815ad0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.705850 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705801    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-certs" pod "b12b03fa-7887-40d9-bcc6-8c71f815ad0b" (UID: "b12b03fa-7887-40d9-bcc6-8c71f815ad0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.705990 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.705945    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istiod-ca-cert" pod "b12b03fa-7887-40d9-bcc6-8c71f815ad0b" (UID: "b12b03fa-7887-40d9-bcc6-8c71f815ad0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:34.706064 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.706010    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.706064 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.706030    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.706064 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.706041    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.706239 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.706069    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.707345 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.707324    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-envoy" pod "b12b03fa-7887-40d9-bcc6-8c71f815ad0b" (UID: "b12b03fa-7887-40d9-bcc6-8c71f815ad0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.707582 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.707568    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-podinfo" pod "b12b03fa-7887-40d9-bcc6-8c71f815ad0b" (UID: "b12b03fa-7887-40d9-bcc6-8c71f815ad0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:34.708135 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.708116    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-kube-api-access-2thqk" pod "b12b03fa-7887-40d9-bcc6-8c71f815ad0b" (UID: "b12b03fa-7887-40d9-bcc6-8c71f815ad0b"). InnerVolumeSpecName "kube-api-access-2thqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:34.708135 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.708130    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-token" pod "b12b03fa-7887-40d9-bcc6-8c71f815ad0b" (UID: "b12b03fa-7887-40d9-bcc6-8c71f815ad0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:34.807131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807103    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-socket\") pod \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") "
Oct 30 20:01:34.807271 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807144    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-certs\") pod \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") "
Oct 30 20:01:34.807271 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807194    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-credential-socket\") pod \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") "
Oct 30 20:01:34.807271 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807236    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-token\" (UniqueName: \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-token\") pod \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") "
Oct 30 20:01:34.807445 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807278    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-kube-api-access-6qwzw\" (UniqueName: \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-kube-api-access-6qwzw\") pod \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") "
Oct 30 20:01:34.807445 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807320    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istiod-ca-cert\") pod \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") "
Oct 30 20:01:34.807445 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807395    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-podinfo\") pod \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") "
Oct 30 20:01:34.807445 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807393    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-socket" pod "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" (UID: "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.807445 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807426    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-envoy\") pod \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") "
Oct 30 20:01:34.807703 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807457    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-data\") pod \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\" (UID: \"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062\") "
Oct 30 20:01:34.807703 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807477    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-certs" pod "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" (UID: "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.807703 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807489    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-credential-socket" pod "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" (UID: "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.807910 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807783    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-data" pod "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" (UID: "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.807910 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.807808    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istiod-ca-cert" pod "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" (UID: "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:34.808147 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.808127    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.808201 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.808156    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.808201 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.808171    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.808201 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.808186    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2thqk\" (UniqueName: \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-kube-api-access-2thqk\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.808290 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.808200    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.808290 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.808215    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.808290 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.808228    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.808290 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.808242    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.808290 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.808256    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b12b03fa-7887-40d9-bcc6-8c71f815ad0b-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.808290 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.808270    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.809442 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.809425    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-token" pod "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" (UID: "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:34.809579 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.809558    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-envoy" pod "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" (UID: "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:34.809999 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.809978    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-podinfo" pod "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" (UID: "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:34.810081 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.809978    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-kube-api-access-6qwzw" pod "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" (UID: "eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062"). InnerVolumeSpecName "kube-api-access-6qwzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:34.909148 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.909125    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.909257 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.909150    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qwzw\" (UniqueName: \"kubernetes.io/projected/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-kube-api-access-6qwzw\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.909257 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.909159    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:34.909257 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:34.909169    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:35.196778 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:01:35.196699    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:01:35.227179 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.227147    4420 generic.go:356] "Generic (PLEG): container finished" podID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerID="e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f" exitCode=0
Oct 30 20:01:35.227371 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.227192    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"
Oct 30 20:01:35.227371 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.227230    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" event={"ID":"b12b03fa-7887-40d9-bcc6-8c71f815ad0b","Type":"ContainerDied","Data":"e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f"}
Oct 30 20:01:35.227371 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.227287    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2" event={"ID":"b12b03fa-7887-40d9-bcc6-8c71f815ad0b","Type":"ContainerDied","Data":"a54e215dffc080ca65ab83ff2930efbbf3fc989767ea625ec900009f4d2d08d4"}
Oct 30 20:01:35.227371 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.227303    4420 scope.go:122] "RemoveContainer" containerID="e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f"
Oct 30 20:01:35.228791 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.228765    4420 generic.go:356] "Generic (PLEG): container finished" podID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerID="317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8" exitCode=0
Oct 30 20:01:35.228925 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.228846    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"
Oct 30 20:01:35.228925 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.228864    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" event={"ID":"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062","Type":"ContainerDied","Data":"317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8"}
Oct 30 20:01:35.228925 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.228908    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4" event={"ID":"eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062","Type":"ContainerDied","Data":"3798ccb7516ce34012d6c9448c8aeb0c5d9aa23efa8bc4ccd12677209cc2e9dc"}
Oct 30 20:01:35.237270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.237248    4420 scope.go:122] "RemoveContainer" containerID="e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f"
Oct 30 20:01:35.237491 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:01:35.237474    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f\": container with ID starting with e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f not found: ID does not exist" containerID="e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f"
Oct 30 20:01:35.237573 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.237499    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f"} err="failed to get container status \"e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f\": rpc error: code = NotFound desc = could not find container \"e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f\": container with ID starting with e9f00ada3a409ca3116fa3242b4c76ca1af9d4f4b61456904b0e38798d9ca76f not found: ID does not exist"
Oct 30 20:01:35.237573 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.237519    4420 scope.go:122] "RemoveContainer" containerID="317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8"
Oct 30 20:01:35.246406 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.246389    4420 scope.go:122] "RemoveContainer" containerID="317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8"
Oct 30 20:01:35.246674 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:01:35.246658    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8\": container with ID starting with 317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8 not found: ID does not exist" containerID="317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8"
Oct 30 20:01:35.246725 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.246681    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8"} err="failed to get container status \"317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8\": rpc error: code = NotFound desc = could not find container \"317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8\": container with ID starting with 317288b3f2b31d33b242f9f8643e48851a8df6db2f081b592437c95a372e00c8 not found: ID does not exist"
Oct 30 20:01:35.402794 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.402761    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"]
Oct 30 20:01:35.422034 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.422010    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-qcpf4"]
Oct 30 20:01:35.542287 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.542224    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"]
Oct 30 20:01:35.560602 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:35.560562    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-dd9b8d79-openshift-default-857b46944f-tbpg2"]
Oct 30 20:01:36.895556 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:36.895527    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" path="/var/lib/kubelet/pods/b12b03fa-7887-40d9-bcc6-8c71f815ad0b/volumes"
Oct 30 20:01:36.896032 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:36.895974    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" path="/var/lib/kubelet/pods/eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062/volumes"
Oct 30 20:01:39.768544 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.768522    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 20:01:39.962634 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.962600    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-credential-socket\") pod \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") "
Oct 30 20:01:39.962865 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.962676    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-socket\") pod \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") "
Oct 30 20:01:39.962865 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.962719    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-data\") pod \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") "
Oct 30 20:01:39.962865 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.962763    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-podinfo\") pod \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") "
Oct 30 20:01:39.963060 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.962877    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-kube-api-access-djxhs\" (UniqueName: \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-kube-api-access-djxhs\") pod \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") "
Oct 30 20:01:39.963060 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.962925    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-certs\") pod \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") "
Oct 30 20:01:39.963060 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.962975    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istiod-ca-cert\") pod \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") "
Oct 30 20:01:39.963060 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.962971    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-credential-socket" pod "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" (UID: "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:39.963060 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.963029    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-envoy\") pod \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") "
Oct 30 20:01:39.963060 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.963049    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-socket" pod "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" (UID: "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:39.963367 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.963067    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-token\" (UniqueName: \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-token\") pod \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\" (UID: \"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85\") "
Oct 30 20:01:39.963367 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.963065    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-data" pod "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" (UID: "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:39.963367 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.963213    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-certs" pod "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" (UID: "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:39.963527 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.963377    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istiod-ca-cert" pod "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" (UID: "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:39.963645 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.963625    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:39.963717 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.963651    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:39.963717 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.963666    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:39.963717 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.963679    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:39.963717 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.963692    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:39.965357 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.965330    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-token" pod "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" (UID: "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:39.965559 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.965539    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-kube-api-access-djxhs" pod "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" (UID: "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85"). InnerVolumeSpecName "kube-api-access-djxhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:39.965728 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.965704    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-envoy" pod "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" (UID: "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:39.965889 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:39.965757    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-podinfo" pod "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" (UID: "ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:40.064574 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.064545    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:40.064574 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.064571    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djxhs\" (UniqueName: \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-kube-api-access-djxhs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:40.064574 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.064581    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:40.064780 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.064590    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:40.268876 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.268748    4420 generic.go:356] "Generic (PLEG): container finished" podID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" containerID="a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab" exitCode=0
Oct 30 20:01:40.268876 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.268845    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"
Oct 30 20:01:40.269126 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.268840    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" event={"ID":"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85","Type":"ContainerDied","Data":"a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab"}
Oct 30 20:01:40.269126 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.269001    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j" event={"ID":"ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85","Type":"ContainerDied","Data":"e6fa39f39cb7251caffb9fc3b8d098197c6d7683b1503960bc859b8c2d333887"}
Oct 30 20:01:40.269126 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.269037    4420 scope.go:122] "RemoveContainer" containerID="a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab"
Oct 30 20:01:40.279825 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.279796    4420 scope.go:122] "RemoveContainer" containerID="a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab"
Oct 30 20:01:40.280119 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:01:40.280098    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab\": container with ID starting with a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab not found: ID does not exist" containerID="a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab"
Oct 30 20:01:40.280194 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.280132    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab"} err="failed to get container status \"a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab\": rpc error: code = NotFound desc = could not find container \"a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab\": container with ID starting with a8f6fab508420524dbad46142b196490b9e90fd5094302d94d3e0262006e07ab not found: ID does not exist"
Oct 30 20:01:40.433242 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.433209    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"]
Oct 30 20:01:40.482188 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.482145    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-686689-openshift-default-5fbf5787bf-zcj9j"]
Oct 30 20:01:40.717231 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.717194    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"]
Oct 30 20:01:40.717437 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.717402    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy" containerID="cri-o://cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885" gracePeriod=30
Oct 30 20:01:40.895501 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:40.895470    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" path="/var/lib/kubelet/pods/ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85/volumes"
Oct 30 20:01:44.722945 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:44.722906    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-6mtks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:01:44.723344 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:44.722975    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:01:46.029548 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.029520    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 20:01:46.031762 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.031741    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-envoy\") pod \"8077dfc1-aa11-4d66-84db-932648ff157f\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") "
Oct 30 20:01:46.031851 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.031784    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-socket\") pod \"8077dfc1-aa11-4d66-84db-932648ff157f\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") "
Oct 30 20:01:46.031851 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.031835    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-certs\") pod \"8077dfc1-aa11-4d66-84db-932648ff157f\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") "
Oct 30 20:01:46.031960 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.031868    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-data\") pod \"8077dfc1-aa11-4d66-84db-932648ff157f\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") "
Oct 30 20:01:46.031960 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.031891    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-istio-token\" (UniqueName: \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-istio-token\") pod \"8077dfc1-aa11-4d66-84db-932648ff157f\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") "
Oct 30 20:01:46.032071 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.031988    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-kube-api-access-j7sh8\" (UniqueName: \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-kube-api-access-j7sh8\") pod \"8077dfc1-aa11-4d66-84db-932648ff157f\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") "
Oct 30 20:01:46.032071 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032009    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8077dfc1-aa11-4d66-84db-932648ff157f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8077dfc1-aa11-4d66-84db-932648ff157f-istio-podinfo\") pod \"8077dfc1-aa11-4d66-84db-932648ff157f\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") "
Oct 30 20:01:46.032071 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032019    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-socket" pod "8077dfc1-aa11-4d66-84db-932648ff157f" (UID: "8077dfc1-aa11-4d66-84db-932648ff157f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:46.032071 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032061    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8077dfc1-aa11-4d66-84db-932648ff157f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8077dfc1-aa11-4d66-84db-932648ff157f-istiod-ca-cert\") pod \"8077dfc1-aa11-4d66-84db-932648ff157f\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") "
Oct 30 20:01:46.032271 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032130    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-credential-socket\") pod \"8077dfc1-aa11-4d66-84db-932648ff157f\" (UID: \"8077dfc1-aa11-4d66-84db-932648ff157f\") "
Oct 30 20:01:46.032271 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032165    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-data" pod "8077dfc1-aa11-4d66-84db-932648ff157f" (UID: "8077dfc1-aa11-4d66-84db-932648ff157f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:46.032271 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032193    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-certs" pod "8077dfc1-aa11-4d66-84db-932648ff157f" (UID: "8077dfc1-aa11-4d66-84db-932648ff157f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:46.032423 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032403    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8077dfc1-aa11-4d66-84db-932648ff157f-istiod-ca-cert" pod "8077dfc1-aa11-4d66-84db-932648ff157f" (UID: "8077dfc1-aa11-4d66-84db-932648ff157f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:46.032532 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032501    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-credential-socket" pod "8077dfc1-aa11-4d66-84db-932648ff157f" (UID: "8077dfc1-aa11-4d66-84db-932648ff157f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:46.032885 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032807    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8077dfc1-aa11-4d66-84db-932648ff157f-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:46.032885 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032859    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:46.032885 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032874    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:46.032885 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032888    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:46.033188 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.032901    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:46.034434 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.034397    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-istio-token" pod "8077dfc1-aa11-4d66-84db-932648ff157f" (UID: "8077dfc1-aa11-4d66-84db-932648ff157f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:46.034514 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.034454    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-envoy" pod "8077dfc1-aa11-4d66-84db-932648ff157f" (UID: "8077dfc1-aa11-4d66-84db-932648ff157f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:46.034661 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.034642    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-kube-api-access-j7sh8" pod "8077dfc1-aa11-4d66-84db-932648ff157f" (UID: "8077dfc1-aa11-4d66-84db-932648ff157f"). InnerVolumeSpecName "kube-api-access-j7sh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:46.034748 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.034733    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8077dfc1-aa11-4d66-84db-932648ff157f-istio-podinfo" pod "8077dfc1-aa11-4d66-84db-932648ff157f" (UID: "8077dfc1-aa11-4d66-84db-932648ff157f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:46.134195 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.134163    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8077dfc1-aa11-4d66-84db-932648ff157f-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:46.134195 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.134195    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:46.134402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.134205    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7sh8\" (UniqueName: \"kubernetes.io/projected/8077dfc1-aa11-4d66-84db-932648ff157f-kube-api-access-j7sh8\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:46.134402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.134214    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8077dfc1-aa11-4d66-84db-932648ff157f-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:46.318607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.318467    4420 generic.go:356] "Generic (PLEG): container finished" podID="8077dfc1-aa11-4d66-84db-932648ff157f" containerID="cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885" exitCode=0
Oct 30 20:01:46.318607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.318521    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"
Oct 30 20:01:46.318607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.318515    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" event={"ID":"8077dfc1-aa11-4d66-84db-932648ff157f","Type":"ContainerDied","Data":"cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885"}
Oct 30 20:01:46.318936 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.318623    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks" event={"ID":"8077dfc1-aa11-4d66-84db-932648ff157f","Type":"ContainerDied","Data":"29a389f56691d1455a5764339db781d72bc776dccfbcb8506846f79b3c830463"}
Oct 30 20:01:46.318936 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.318665    4420 scope.go:122] "RemoveContainer" containerID="cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885"
Oct 30 20:01:46.331446 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.331422    4420 scope.go:122] "RemoveContainer" containerID="cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885"
Oct 30 20:01:46.331762 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:01:46.331743    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885\": container with ID starting with cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885 not found: ID does not exist" containerID="cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885"
Oct 30 20:01:46.331886 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.331770    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885"} err="failed to get container status \"cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885\": rpc error: code = NotFound desc = could not find container \"cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885\": container with ID starting with cc084cc5d686288851d8f5b1d5944a79e39d3083fe7888491a77e2dce7a48885 not found: ID does not exist"
Oct 30 20:01:46.490430 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.490398    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"]
Oct 30 20:01:46.493999 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.493978    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-6mtks"]
Oct 30 20:01:46.895376 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:46.895345    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" path="/var/lib/kubelet/pods/8077dfc1-aa11-4d66-84db-932648ff157f/volumes"
Oct 30 20:01:47.416175 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:47.416133    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"]
Oct 30 20:01:47.416595 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:47.416335    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerName="istio-proxy" containerID="cri-o://059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865" gracePeriod=30
Oct 30 20:01:52.680476 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.680451    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:01:52.711882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.711770    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-kube-api-access-gn7bl\" (UniqueName: \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-kube-api-access-gn7bl\") pod \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") "
Oct 30 20:01:52.711882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.711846    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-credential-socket\") pod \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") "
Oct 30 20:01:52.711882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.711870    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-podinfo\") pod \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") "
Oct 30 20:01:52.712162 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.711900    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istiod-ca-cert\") pod \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") "
Oct 30 20:01:52.712162 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.712002    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-socket\") pod \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") "
Oct 30 20:01:52.712162 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.712061    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-token\" (UniqueName: \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-token\") pod \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") "
Oct 30 20:01:52.712162 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.712100    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-data\") pod \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") "
Oct 30 20:01:52.712162 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.712155    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-certs\") pod \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") "
Oct 30 20:01:52.712162 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.712153    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-credential-socket" pod "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" (UID: "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:52.712457 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.712185    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-envoy\") pod \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\" (UID: \"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e\") "
Oct 30 20:01:52.712513 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.712480    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-socket" pod "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" (UID: "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:52.712724 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.712670    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.712724 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.712684    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-certs" pod "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" (UID: "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:52.712971 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.712696    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.712971 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.712951    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-data" pod "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" (UID: "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:52.713197 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.713173    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istiod-ca-cert" pod "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" (UID: "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:52.714485 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.714453    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-podinfo" pod "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" (UID: "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:01:52.714592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.714517    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-token" pod "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" (UID: "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:52.714650 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.714588    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-kube-api-access-gn7bl" pod "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" (UID: "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e"). InnerVolumeSpecName "kube-api-access-gn7bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:52.714706 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.714670    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-envoy" pod "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" (UID: "be2702ec-d3bf-45f9-a6cd-cfff1b296d0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:01:52.814349 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.814311    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn7bl\" (UniqueName: \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-kube-api-access-gn7bl\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.814349 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.814342    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.814349 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.814352    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.814349 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.814360    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.814628 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.814369    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.814628 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.814377    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:52.814628 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:52.814385    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:53.361439 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:53.361403    4420 generic.go:356] "Generic (PLEG): container finished" podID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerID="059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865" exitCode=0
Oct 30 20:01:53.361618 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:53.361466    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"
Oct 30 20:01:53.361618 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:53.361482    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" event={"ID":"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e","Type":"ContainerDied","Data":"059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865"}
Oct 30 20:01:53.361618 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:53.361545    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2" event={"ID":"be2702ec-d3bf-45f9-a6cd-cfff1b296d0e","Type":"ContainerDied","Data":"8502cda12158d1ffcecdecaeff0ce0a057959da09f88618bb87e7a06bafaf19e"}
Oct 30 20:01:53.361618 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:53.361570    4420 scope.go:122] "RemoveContainer" containerID="059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865"
Oct 30 20:01:53.371724 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:53.371701    4420 scope.go:122] "RemoveContainer" containerID="059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865"
Oct 30 20:01:53.372069 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:01:53.372045    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865\": container with ID starting with 059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865 not found: ID does not exist" containerID="059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865"
Oct 30 20:01:53.372180 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:53.372076    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865"} err="failed to get container status \"059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865\": rpc error: code = NotFound desc = could not find container \"059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865\": container with ID starting with 059098699edbc011c65d9a58e05b873a2097f4bf87199e9ab115523d08603865 not found: ID does not exist"
Oct 30 20:01:53.527470 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:53.527436    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"]
Oct 30 20:01:53.532327 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:53.532300    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f55cd-openshift-default-9dc497cb5-54sj2"]
Oct 30 20:01:54.895144 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:54.895109    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" path="/var/lib/kubelet/pods/be2702ec-d3bf-45f9-a6cd-cfff1b296d0e/volumes"
Oct 30 20:01:57.328502 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.328468    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"]
Oct 30 20:01:57.329047 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.328682    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f" podUID="82260772-1e21-46a5-bc36-4c4cc7ec8f75" containerName="maas-api" containerID="cri-o://bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274" gracePeriod=30
Oct 30 20:01:57.581731 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.581670    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 20:01:57.667783 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.667747    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-metrics-certs\" (UniqueName: \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-metrics-certs\") pod \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") "
Oct 30 20:01:57.667974 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.667944    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-maas-api-tls\") pod \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") "
Oct 30 20:01:57.668013 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.667982    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82260772-1e21-46a5-bc36-4c4cc7ec8f75-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82260772-1e21-46a5-bc36-4c4cc7ec8f75-openshift-service-ca\") pod \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") "
Oct 30 20:01:57.668048 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.668019    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82260772-1e21-46a5-bc36-4c4cc7ec8f75-kube-api-access-gwbdd\" (UniqueName: \"kubernetes.io/projected/82260772-1e21-46a5-bc36-4c4cc7ec8f75-kube-api-access-gwbdd\") pod \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\" (UID: \"82260772-1e21-46a5-bc36-4c4cc7ec8f75\") "
Oct 30 20:01:57.668361 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.668327    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82260772-1e21-46a5-bc36-4c4cc7ec8f75-openshift-service-ca" pod "82260772-1e21-46a5-bc36-4c4cc7ec8f75" (UID: "82260772-1e21-46a5-bc36-4c4cc7ec8f75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:01:57.670035 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.670010    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-metrics-certs" pod "82260772-1e21-46a5-bc36-4c4cc7ec8f75" (UID: "82260772-1e21-46a5-bc36-4c4cc7ec8f75"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:57.670184 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.670163    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-maas-api-tls" pod "82260772-1e21-46a5-bc36-4c4cc7ec8f75" (UID: "82260772-1e21-46a5-bc36-4c4cc7ec8f75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:01:57.670253 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.670179    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82260772-1e21-46a5-bc36-4c4cc7ec8f75-kube-api-access-gwbdd" pod "82260772-1e21-46a5-bc36-4c4cc7ec8f75" (UID: "82260772-1e21-46a5-bc36-4c4cc7ec8f75"). InnerVolumeSpecName "kube-api-access-gwbdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:01:57.769537 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.769506    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:57.769537 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.769536    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82260772-1e21-46a5-bc36-4c4cc7ec8f75-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:57.769537 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.769546    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwbdd\" (UniqueName: \"kubernetes.io/projected/82260772-1e21-46a5-bc36-4c4cc7ec8f75-kube-api-access-gwbdd\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:57.769780 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:57.769556    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82260772-1e21-46a5-bc36-4c4cc7ec8f75-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:01:58.388162 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:58.388125    4420 generic.go:356] "Generic (PLEG): container finished" podID="82260772-1e21-46a5-bc36-4c4cc7ec8f75" containerID="bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274" exitCode=0
Oct 30 20:01:58.388667 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:58.388185    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"
Oct 30 20:01:58.388667 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:58.388213    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f" event={"ID":"82260772-1e21-46a5-bc36-4c4cc7ec8f75","Type":"ContainerDied","Data":"bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274"}
Oct 30 20:01:58.388667 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:58.388281    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f" event={"ID":"82260772-1e21-46a5-bc36-4c4cc7ec8f75","Type":"ContainerDied","Data":"57133b61d155148113d86d0e8a91df480cc5bed2b6aba34f326d578f478c36da"}
Oct 30 20:01:58.388667 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:58.388310    4420 scope.go:122] "RemoveContainer" containerID="bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274"
Oct 30 20:01:58.398954 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:58.398934    4420 scope.go:122] "RemoveContainer" containerID="bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274"
Oct 30 20:01:58.399237 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:01:58.399218    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274\": container with ID starting with bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274 not found: ID does not exist" containerID="bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274"
Oct 30 20:01:58.399310 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:58.399246    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274"} err="failed to get container status \"bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274\": rpc error: code = NotFound desc = could not find container \"bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274\": container with ID starting with bd22724e7cbb2e59bc0dd314dcee17ff6dc0efbc18a5e6ad8cfb52a1e7fcb274 not found: ID does not exist"
Oct 30 20:01:58.552329 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:58.552298    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"]
Oct 30 20:01:58.554331 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:58.554305    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2d9861-6978b9f7c8-76d4f"]
Oct 30 20:01:58.895317 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:01:58.895274    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82260772-1e21-46a5-bc36-4c4cc7ec8f75" path="/var/lib/kubelet/pods/82260772-1e21-46a5-bc36-4c4cc7ec8f75/volumes"
Oct 30 20:02:02.086493 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:02.086460    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"]
Oct 30 20:02:02.135328 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:02.135251    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"]
Oct 30 20:02:05.196883 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:02:05.196840    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:02:06.478570 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:06.478534    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"]
Oct 30 20:02:06.479034 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:06.478739    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy" containerID="cri-o://c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae" gracePeriod=30
Oct 30 20:02:09.106090 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:09.106050    4420 patch_prober.go:29] interesting pod/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 20:02:09.106570 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:09.106123    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 20:02:11.002901 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.002869    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"]
Oct 30 20:02:11.003333 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003319    4420 cpu_manager.go:414] "containerMap: removing container" podUID="dab3dae4-c6eb-4256-95a1-c5354500fba4" containerName="maas-api"
Oct 30 20:02:11.003376 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003335    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab3dae4-c6eb-4256-95a1-c5354500fba4" containerName="maas-api"
Oct 30 20:02:11.003376 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003349    4420 cpu_manager.go:414] "containerMap: removing container" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerName="istio-proxy"
Oct 30 20:02:11.003376 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003353    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerName="istio-proxy"
Oct 30 20:02:11.003376 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003361    4420 cpu_manager.go:414] "containerMap: removing container" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerName="istio-proxy"
Oct 30 20:02:11.003376 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003365    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerName="istio-proxy"
Oct 30 20:02:11.003376 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003372    4420 cpu_manager.go:414] "containerMap: removing container" podUID="123635db-8c61-493e-b18d-e17afa4203bc" containerName="maas-api"
Oct 30 20:02:11.003376 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003376    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="123635db-8c61-493e-b18d-e17afa4203bc" containerName="maas-api"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003389    4420 cpu_manager.go:414] "containerMap: removing container" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003393    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003402    4420 cpu_manager.go:414] "containerMap: removing container" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003408    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003415    4420 cpu_manager.go:414] "containerMap: removing container" podUID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" containerName="istio-proxy"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003419    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" containerName="istio-proxy"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003430    4420 cpu_manager.go:414] "containerMap: removing container" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerName="istio-proxy"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003434    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerName="istio-proxy"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003452    4420 cpu_manager.go:414] "containerMap: removing container" podUID="82260772-1e21-46a5-bc36-4c4cc7ec8f75" containerName="maas-api"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003456    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82260772-1e21-46a5-bc36-4c4cc7ec8f75" containerName="maas-api"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003515    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="82260772-1e21-46a5-bc36-4c4cc7ec8f75" containerName="maas-api"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003523    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab3dae4-c6eb-4256-95a1-c5354500fba4" containerName="maas-api"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003529    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b12b03fa-7887-40d9-bcc6-8c71f815ad0b" containerName="istio-proxy"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003533    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2702ec-d3bf-45f9-a6cd-cfff1b296d0e" containerName="istio-proxy"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003539    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="123635db-8c61-493e-b18d-e17afa4203bc" containerName="maas-api"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003545    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44abbb2-14b5-4553-b03d-e29000457db2" containerName="main"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003551    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8077dfc1-aa11-4d66-84db-932648ff157f" containerName="istio-proxy"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003557    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec2a09c8-fe31-46a9-9cf3-1c2b74d45b85" containerName="istio-proxy"
Oct 30 20:02:11.003592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.003596    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb886fa6-4fe0-47e5-9cd1-a1ec2ee6b062" containerName="istio-proxy"
Oct 30 20:02:11.006518 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.006502    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.109963 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.109932    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"]
Oct 30 20:02:11.113973 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.113942    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.114119 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.114009    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4ca4ee-283d-4613-8d97-09478295784d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.114192 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.114170    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.114257 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.114200    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.114296 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.114272    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nz8v\" (UniqueName: \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-kube-api-access-4nz8v\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.114351 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.114324    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.114434 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.114418    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.114491 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.114478    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.114562 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.114546    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4ca4ee-283d-4613-8d97-09478295784d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.215764 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.215724    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.215980 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.215778    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4ca4ee-283d-4613-8d97-09478295784d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.215980 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.215842    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.215980 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.215858    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.215980 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.215891    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nz8v\" (UniqueName: \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-kube-api-access-4nz8v\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.216181 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.216035    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.216181 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.216107    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.216181 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.216163    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.216316 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.216200    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4ca4ee-283d-4613-8d97-09478295784d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.216316 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.216231    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.216316 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.216289    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.216518 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.216498    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.216577 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.216528    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.216885 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.216868    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4ca4ee-283d-4613-8d97-09478295784d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.218938 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.218915    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.219233 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.219208    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4ca4ee-283d-4613-8d97-09478295784d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.273513 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.273438    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.273697 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.273630    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nz8v\" (UniqueName: \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-kube-api-access-4nz8v\") pod \"maas-default-gateway-openshift-default-6db586dd6c-8r4gb\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.319710 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.319668    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:11.715398 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.715373    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 20:02:11.751210 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.751184    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"]
Oct 30 20:02:11.753209 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:02:11.753180    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4ca4ee_283d_4613_8d97_09478295784d.slice/crio-456d5a961953e1e5baa8f663229186e5a63d1ba91039f7ab55abcbeb8e9802fb WatchSource:0}: Error finding container 456d5a961953e1e5baa8f663229186e5a63d1ba91039f7ab55abcbeb8e9802fb: Status 404 returned error can't find the container with id 456d5a961953e1e5baa8f663229186e5a63d1ba91039f7ab55abcbeb8e9802fb
Oct 30 20:02:11.755573 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.755543    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:11.755639 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.755608    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:11.755639 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.755635    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:11.823421 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.823392    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-credential-socket\") pod \"a5a35842-06c7-43f1-b1b5-cdabd86db545\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") "
Oct 30 20:02:11.823607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.823442    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-token\" (UniqueName: \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-token\") pod \"a5a35842-06c7-43f1-b1b5-cdabd86db545\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") "
Oct 30 20:02:11.823607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.823471    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-data\") pod \"a5a35842-06c7-43f1-b1b5-cdabd86db545\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") "
Oct 30 20:02:11.823607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.823504    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-podinfo\") pod \"a5a35842-06c7-43f1-b1b5-cdabd86db545\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") "
Oct 30 20:02:11.823607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.823589    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-envoy\") pod \"a5a35842-06c7-43f1-b1b5-cdabd86db545\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") "
Oct 30 20:02:11.823828 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.823700    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-credential-socket" pod "a5a35842-06c7-43f1-b1b5-cdabd86db545" (UID: "a5a35842-06c7-43f1-b1b5-cdabd86db545"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:11.823828 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.823711    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a35842-06c7-43f1-b1b5-cdabd86db545-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a35842-06c7-43f1-b1b5-cdabd86db545-istiod-ca-cert\") pod \"a5a35842-06c7-43f1-b1b5-cdabd86db545\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") "
Oct 30 20:02:11.823949 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.823807    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-socket\") pod \"a5a35842-06c7-43f1-b1b5-cdabd86db545\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") "
Oct 30 20:02:11.823949 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.823849    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-data" pod "a5a35842-06c7-43f1-b1b5-cdabd86db545" (UID: "a5a35842-06c7-43f1-b1b5-cdabd86db545"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:11.823949 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.823895    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-kube-api-access-jrjr7\" (UniqueName: \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-kube-api-access-jrjr7\") pod \"a5a35842-06c7-43f1-b1b5-cdabd86db545\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") "
Oct 30 20:02:11.824108 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.823949    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-certs\") pod \"a5a35842-06c7-43f1-b1b5-cdabd86db545\" (UID: \"a5a35842-06c7-43f1-b1b5-cdabd86db545\") "
Oct 30 20:02:11.824170 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.824136    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a35842-06c7-43f1-b1b5-cdabd86db545-istiod-ca-cert" pod "a5a35842-06c7-43f1-b1b5-cdabd86db545" (UID: "a5a35842-06c7-43f1-b1b5-cdabd86db545"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:11.824232 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.824186    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-socket" pod "a5a35842-06c7-43f1-b1b5-cdabd86db545" (UID: "a5a35842-06c7-43f1-b1b5-cdabd86db545"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:11.824372 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.824344    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-certs" pod "a5a35842-06c7-43f1-b1b5-cdabd86db545" (UID: "a5a35842-06c7-43f1-b1b5-cdabd86db545"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:11.824506 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.824487    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a35842-06c7-43f1-b1b5-cdabd86db545-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:11.824573 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.824512    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:11.824573 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.824526    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:11.824694 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.824586    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:11.824751 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.824711    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:11.825989 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.825964    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-envoy" pod "a5a35842-06c7-43f1-b1b5-cdabd86db545" (UID: "a5a35842-06c7-43f1-b1b5-cdabd86db545"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:11.826266 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.826111    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-kube-api-access-jrjr7" pod "a5a35842-06c7-43f1-b1b5-cdabd86db545" (UID: "a5a35842-06c7-43f1-b1b5-cdabd86db545"). InnerVolumeSpecName "kube-api-access-jrjr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:11.826266 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.826245    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-podinfo" pod "a5a35842-06c7-43f1-b1b5-cdabd86db545" (UID: "a5a35842-06c7-43f1-b1b5-cdabd86db545"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:02:11.826447 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.826432    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-token" pod "a5a35842-06c7-43f1-b1b5-cdabd86db545" (UID: "a5a35842-06c7-43f1-b1b5-cdabd86db545"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:11.926219 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.926178    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrjr7\" (UniqueName: \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-kube-api-access-jrjr7\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:11.926219 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.926215    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:11.926219 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.926229    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:11.926482 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:11.926243    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a35842-06c7-43f1-b1b5-cdabd86db545-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:12.489268 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.489167    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" event={"ID":"ca4ca4ee-283d-4613-8d97-09478295784d","Type":"ContainerStarted","Data":"4f6f5d2f0d1f3ad62bb179d49d09bcec881bd59709f6f351ac32c0377f171cfc"}
Oct 30 20:02:12.489268 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.489237    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" event={"ID":"ca4ca4ee-283d-4613-8d97-09478295784d","Type":"ContainerStarted","Data":"456d5a961953e1e5baa8f663229186e5a63d1ba91039f7ab55abcbeb8e9802fb"}
Oct 30 20:02:12.490960 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.490934    4420 generic.go:356] "Generic (PLEG): container finished" podID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerID="c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae" exitCode=0
Oct 30 20:02:12.491083 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.490979    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"
Oct 30 20:02:12.491083 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.491003    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" event={"ID":"a5a35842-06c7-43f1-b1b5-cdabd86db545","Type":"ContainerDied","Data":"c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae"}
Oct 30 20:02:12.491083 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.491052    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb" event={"ID":"a5a35842-06c7-43f1-b1b5-cdabd86db545","Type":"ContainerDied","Data":"b5eba0aa24a2cddda41c447af41965dd7a03d02020e96b780ccb960ea513ce5f"}
Oct 30 20:02:12.491251 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.491081    4420 scope.go:122] "RemoveContainer" containerID="c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae"
Oct 30 20:02:12.501114 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.501095    4420 scope.go:122] "RemoveContainer" containerID="161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e"
Oct 30 20:02:12.509685 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.509664    4420 scope.go:122] "RemoveContainer" containerID="c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae"
Oct 30 20:02:12.509987 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:02:12.509967    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae\": container with ID starting with c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae not found: ID does not exist" containerID="c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae"
Oct 30 20:02:12.510062 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.509998    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae"} err="failed to get container status \"c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae\": rpc error: code = NotFound desc = could not find container \"c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae\": container with ID starting with c7b0e931238f33e2d8b9eb905a05b49b9cf84e517638c5f827b9e0ea675672ae not found: ID does not exist"
Oct 30 20:02:12.510062 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.510018    4420 scope.go:122] "RemoveContainer" containerID="161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e"
Oct 30 20:02:12.510307 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:02:12.510284    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e\": container with ID starting with 161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e not found: ID does not exist" containerID="161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e"
Oct 30 20:02:12.510366 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.510316    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e"} err="failed to get container status \"161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e\": rpc error: code = NotFound desc = could not find container \"161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e\": container with ID starting with 161baf9299f15084e3a21b62f905743e39404656d05dbbdf0330fc91aa86ed8e not found: ID does not exist"
Oct 30 20:02:12.609282 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.609225    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" podStartSLOduration=2.609210774 podStartE2EDuration="2.609210774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:12.606533999 +0000 UTC m=+2779.817751170" watchObservedRunningTime="2026-10-30 20:02:12.609210774 +0000 UTC m=+2779.820427918"
Oct 30 20:02:12.721421 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.721383    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"]
Oct 30 20:02:12.725433 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.725406    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-2d9861-openshift-default-58ddf574b9-88jdb"]
Oct 30 20:02:12.896352 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:12.896320    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" path="/var/lib/kubelet/pods/a5a35842-06c7-43f1-b1b5-cdabd86db545/volumes"
Oct 30 20:02:13.291093 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.290995    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"]
Oct 30 20:02:13.291502 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.291487    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy"
Oct 30 20:02:13.291565 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.291504    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy"
Oct 30 20:02:13.291624 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.291596    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy"
Oct 30 20:02:13.294866 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.294838    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.320176 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.320146    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:13.322160 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.322128    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8r4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:13.322302 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.322184    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" podUID="ca4ca4ee-283d-4613-8d97-09478295784d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:13.350989 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.350962    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"]
Oct 30 20:02:13.352133 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.352101    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-132119\"/\"openshift-service-ca.crt\""
Oct 30 20:02:13.382852 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.382171    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-132119\"/\"kube-root-ca.crt\""
Oct 30 20:02:13.397496 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.397468    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-132119\"/\"default-dockercfg-mg4wt\""
Oct 30 20:02:13.397496 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.397468    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-132119\"/\"test-model-132119-kserve-self-signed-certs\""
Oct 30 20:02:13.443509 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.443476    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tls-certs\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.443703 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.443546    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tmp-dir\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.443703 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.443612    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd7bp\" (UniqueName: \"kubernetes.io/projected/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-kube-api-access-kd7bp\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.443788 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.443757    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-dshm\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.444416 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.444352    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-home\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.444606 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.444431    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-model-cache\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.545683 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.545590    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-model-cache\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.545683 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.545636    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tls-certs\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.545683 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.545674    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tmp-dir\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.546361 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.545690    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd7bp\" (UniqueName: \"kubernetes.io/projected/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-kube-api-access-kd7bp\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.546361 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.545738    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-dshm\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.546361 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.545846    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-home\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.546361 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.546032    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-model-cache\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.546361 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.546308    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tmp-dir\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.546566 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.546379    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-home\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.548210 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.548187    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-dshm\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.548389 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.548372    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tls-certs\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.603401 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.603363    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd7bp\" (UniqueName: \"kubernetes.io/projected/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-kube-api-access-kd7bp\") pod \"test-model-132119-kserve-6b5d445fdf-6x7jg\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") " pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.606065 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.606040    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:13.855192 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:13.855099    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"]
Oct 30 20:02:14.004844 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:14.004803    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"]
Oct 30 20:02:14.006727 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:02:14.006701    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda52f1425_a774_4d44_aa7c_1b0ddb750fb9.slice/crio-99c6eeb6d81c6d447a896a19aab9c20d953da1210126b83c3ab2485045d223dd WatchSource:0}: Error finding container 99c6eeb6d81c6d447a896a19aab9c20d953da1210126b83c3ab2485045d223dd: Status 404 returned error can't find the container with id 99c6eeb6d81c6d447a896a19aab9c20d953da1210126b83c3ab2485045d223dd
Oct 30 20:02:14.320740 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:14.320699    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8r4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:14.320949 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:14.320750    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" podUID="ca4ca4ee-283d-4613-8d97-09478295784d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:14.504097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:14.504056    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg" event={"ID":"a52f1425-a774-4d44-aa7c-1b0ddb750fb9","Type":"ContainerStarted","Data":"165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19"}
Oct 30 20:02:14.504308 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:14.504106    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg" event={"ID":"a52f1425-a774-4d44-aa7c-1b0ddb750fb9","Type":"ContainerStarted","Data":"99c6eeb6d81c6d447a896a19aab9c20d953da1210126b83c3ab2485045d223dd"}
Oct 30 20:02:14.606403 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:14.606322    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:14.618081 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:14.618035    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg" podStartSLOduration=1.618022539 podStartE2EDuration="1.618022539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:14.615725275 +0000 UTC m=+2781.826942420" watchObservedRunningTime="2026-10-30 20:02:14.618022539 +0000 UTC m=+2781.829239682"
Oct 30 20:02:15.320583 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:15.320552    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8r4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:15.320781 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:15.320601    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" podUID="ca4ca4ee-283d-4613-8d97-09478295784d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:16.320940 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:16.320902    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8r4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:16.321375 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:16.320967    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" podUID="ca4ca4ee-283d-4613-8d97-09478295784d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:17.108208 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.108169    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" containerID="cri-o://59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42" gracePeriod=45
Oct 30 20:02:17.155885 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.155845    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main" containerID="cri-o://67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6" gracePeriod=45
Oct 30 20:02:17.321110 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.321066    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": dial tcp 10.133.0.55:8000: connect: connection refused"
Oct 30 20:02:17.321110 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.321095    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8r4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:17.321653 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.321133    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" podUID="ca4ca4ee-283d-4613-8d97-09478295784d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:17.498122 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.498049    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 20:02:17.515207 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.515181    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 20:02:17.515804 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.515783    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-kube-api-access-zm27j\" (UniqueName: \"kubernetes.io/projected/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-kube-api-access-zm27j\") pod \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") "
Oct 30 20:02:17.515936 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.515857    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-home\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-home\") pod \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") "
Oct 30 20:02:17.515936 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.515888    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-home\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-home\") pod \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") "
Oct 30 20:02:17.515936 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.515933    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tmp-dir\") pod \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") "
Oct 30 20:02:17.516093 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.515960    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-dshm\") pod \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") "
Oct 30 20:02:17.516093 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.515986    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tmp-dir\") pod \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") "
Oct 30 20:02:17.516093 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.516017    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tls-certs\" (UniqueName: \"kubernetes.io/secret/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tls-certs\") pod \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") "
Oct 30 20:02:17.516093 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.516060    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-kube-api-access-lkt44\" (UniqueName: \"kubernetes.io/projected/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-kube-api-access-lkt44\") pod \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") "
Oct 30 20:02:17.516270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.516107    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tls-certs\") pod \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") "
Oct 30 20:02:17.516270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.516240    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-model-cache\") pod \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\" (UID: \"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8\") "
Oct 30 20:02:17.516362 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.516286    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-dshm\") pod \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") "
Oct 30 20:02:17.516450 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.516424    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-model-cache\") pod \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\" (UID: \"a934ee61-e753-4b2a-8fc5-2e9d84698ca3\") "
Oct 30 20:02:17.517144 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.517119    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-model-cache" pod "a934ee61-e753-4b2a-8fc5-2e9d84698ca3" (UID: "a934ee61-e753-4b2a-8fc5-2e9d84698ca3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.517381 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.517360    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tmp-dir" pod "a934ee61-e753-4b2a-8fc5-2e9d84698ca3" (UID: "a934ee61-e753-4b2a-8fc5-2e9d84698ca3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.519582 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.519557    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-dshm" pod "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" (UID: "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.520585 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.520556    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tls-certs" pod "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" (UID: "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:17.521052 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.521033    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-model-cache" pod "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" (UID: "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.521127 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.521060    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-home" pod "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" (UID: "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.521270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.521254    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-home" pod "a934ee61-e753-4b2a-8fc5-2e9d84698ca3" (UID: "a934ee61-e753-4b2a-8fc5-2e9d84698ca3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.521476 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.521450    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tmp-dir" pod "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" (UID: "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.521671 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.521645    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-kube-api-access-zm27j" pod "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" (UID: "b6167cb2-b4ba-4b88-9ce4-fc2636006ec8"). InnerVolumeSpecName "kube-api-access-zm27j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:17.523344 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.523314    4420 generic.go:356] "Generic (PLEG): container finished" podID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerID="59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42" exitCode=0
Oct 30 20:02:17.523513 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.523499    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"
Oct 30 20:02:17.523575 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.523552    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" event={"ID":"a934ee61-e753-4b2a-8fc5-2e9d84698ca3","Type":"ContainerDied","Data":"59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42"}
Oct 30 20:02:17.523630 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.523587    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b" event={"ID":"a934ee61-e753-4b2a-8fc5-2e9d84698ca3","Type":"ContainerDied","Data":"31b10c627ed1d43290fa8f3274720c37690ff4cc3856fe29e7d3b411b6ffff14"}
Oct 30 20:02:17.523630 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.523608    4420 scope.go:122] "RemoveContainer" containerID="59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42"
Oct 30 20:02:17.526367 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.526341    4420 generic.go:356] "Generic (PLEG): container finished" podID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerID="67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6" exitCode=0
Oct 30 20:02:17.526445 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.526381    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" event={"ID":"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8","Type":"ContainerDied","Data":"67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6"}
Oct 30 20:02:17.526445 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.526411    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt" event={"ID":"b6167cb2-b4ba-4b88-9ce4-fc2636006ec8","Type":"ContainerDied","Data":"2bf72a1d4a30857c8a2b4157988bcec96db66552c80d397966e8e127f55de3e8"}
Oct 30 20:02:17.526548 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.526472    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"
Oct 30 20:02:17.531004 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.530977    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-kube-api-access-lkt44" pod "a934ee61-e753-4b2a-8fc5-2e9d84698ca3" (UID: "a934ee61-e753-4b2a-8fc5-2e9d84698ca3"). InnerVolumeSpecName "kube-api-access-lkt44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:17.531102 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.531065    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-dshm" pod "a934ee61-e753-4b2a-8fc5-2e9d84698ca3" (UID: "a934ee61-e753-4b2a-8fc5-2e9d84698ca3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:17.533100 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.532549    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tls-certs" pod "a934ee61-e753-4b2a-8fc5-2e9d84698ca3" (UID: "a934ee61-e753-4b2a-8fc5-2e9d84698ca3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:17.534921 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.534902    4420 scope.go:122] "RemoveContainer" containerID="59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42"
Oct 30 20:02:17.535215 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:02:17.535194    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42\": container with ID starting with 59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42 not found: ID does not exist" containerID="59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42"
Oct 30 20:02:17.535270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.535226    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42"} err="failed to get container status \"59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42\": rpc error: code = NotFound desc = could not find container \"59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42\": container with ID starting with 59a27ea4d82036c62d52c1a954f17d01c8b872594dc7141b0bb828fc21ed3b42 not found: ID does not exist"
Oct 30 20:02:17.535270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.535250    4420 scope.go:122] "RemoveContainer" containerID="67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6"
Oct 30 20:02:17.544661 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.544638    4420 scope.go:122] "RemoveContainer" containerID="67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6"
Oct 30 20:02:17.545379 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:02:17.545350    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6\": container with ID starting with 67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6 not found: ID does not exist" containerID="67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6"
Oct 30 20:02:17.545478 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.545382    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6"} err="failed to get container status \"67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6\": rpc error: code = NotFound desc = could not find container \"67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6\": container with ID starting with 67e793a7daeb682bfd383de0e87da175cf8f3fb533a01263b3f4fe5c363b09e6 not found: ID does not exist"
Oct 30 20:02:17.617523 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617497    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.617523 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617521    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.617523 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617531    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.617752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617541    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm27j\" (UniqueName: \"kubernetes.io/projected/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-kube-api-access-zm27j\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.617752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617549    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.617752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617557    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.617752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617564    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.617752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617571    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.617752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617578    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.617752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617585    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.617752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617592    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkt44\" (UniqueName: \"kubernetes.io/projected/a934ee61-e753-4b2a-8fc5-2e9d84698ca3-kube-api-access-lkt44\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.617752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.617599    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:17.797341 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.797263    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"]
Oct 30 20:02:17.806956 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.800587    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/premium-w2-a3f9b7-kserve-85767776db-7lpdt"]
Oct 30 20:02:17.961296 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.961265    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"]
Oct 30 20:02:17.967236 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:17.967208    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a3f9b7/distinct-w2-a3f9b7-kserve-688fd9997c-hrm9b"]
Oct 30 20:02:18.321055 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:18.321019    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8r4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:18.321237 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:18.321074    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" podUID="ca4ca4ee-283d-4613-8d97-09478295784d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:18.608092 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:18.608059    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"]
Oct 30 20:02:18.896801 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:18.896760    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" path="/var/lib/kubelet/pods/a934ee61-e753-4b2a-8fc5-2e9d84698ca3/volumes"
Oct 30 20:02:18.897291 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:18.897272    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" path="/var/lib/kubelet/pods/b6167cb2-b4ba-4b88-9ce4-fc2636006ec8/volumes"
Oct 30 20:02:19.320321 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:19.320233    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8r4gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 30 20:02:19.320321 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:19.320302    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" podUID="ca4ca4ee-283d-4613-8d97-09478295784d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 30 20:02:20.511050 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:20.511020    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:20.512055 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:20.512026    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:20.512960 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:20.512933    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:02:23.607179 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:23.607144    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:23.608577 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:23.608546    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh" podUID="c21754ed-109f-45af-ad88-de64f423c0fc" containerName="payload-pre-processing" containerID="cri-o://5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9" gracePeriod=25
Oct 30 20:02:23.619961 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:23.619939    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:24.583181 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:24.583149    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:02:25.973013 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:25.972976    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"]
Oct 30 20:02:25.973445 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:25.973432    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main"
Oct 30 20:02:25.973486 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:25.973446    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main"
Oct 30 20:02:25.973486 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:25.973458    4420 cpu_manager.go:414] "containerMap: removing container" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main"
Oct 30 20:02:25.973486 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:25.973463    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main"
Oct 30 20:02:25.973486 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:25.973473    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy"
Oct 30 20:02:25.973486 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:25.973479    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy"
Oct 30 20:02:25.973649 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:25.973561    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a35842-06c7-43f1-b1b5-cdabd86db545" containerName="istio-proxy"
Oct 30 20:02:25.973649 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:25.973573    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a934ee61-e753-4b2a-8fc5-2e9d84698ca3" containerName="main"
Oct 30 20:02:25.973649 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:25.973579    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6167cb2-b4ba-4b88-9ce4-fc2636006ec8" containerName="main"
Oct 30 20:02:25.975581 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:25.975565    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.005062 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.005026    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.005256 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.005074    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.005256 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.005191    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442ef4a0-7909-42c2-8567-8d7555361aee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.005256 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.005234    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.005256 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.005251    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.005461 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.005355    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.005461 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.005399    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwcn\" (UniqueName: \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-kube-api-access-mgwcn\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.005525 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.005467    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.005525 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.005485    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442ef4a0-7909-42c2-8567-8d7555361aee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.083472 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.083438    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"]
Oct 30 20:02:26.106429 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.106396    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.106429 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.106428    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.106678 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.106456    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.106678 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.106497    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwcn\" (UniqueName: \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-kube-api-access-mgwcn\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.106678 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.106541    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.106678 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.106565    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442ef4a0-7909-42c2-8567-8d7555361aee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.106678 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.106627    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.106678 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.106670    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.107034 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.106740    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442ef4a0-7909-42c2-8567-8d7555361aee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.107034 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.106921    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.107220 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.107192    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.107350 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.107313    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.107421 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.107349    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.107765 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.107739    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442ef4a0-7909-42c2-8567-8d7555361aee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.109475 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.109454    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.109626 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.109563    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442ef4a0-7909-42c2-8567-8d7555361aee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.165689 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.165652    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwcn\" (UniqueName: \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-kube-api-access-mgwcn\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.166022 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.166003    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-748lj\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.287349 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.287268    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:26.710689 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.710629    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"]
Oct 30 20:02:26.713074 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:02:26.713047    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442ef4a0_7909_42c2_8567_8d7555361aee.slice/crio-f28f9536b15d44898b61bf31197c0b710da91f58865510b7b5522432e025c337 WatchSource:0}: Error finding container f28f9536b15d44898b61bf31197c0b710da91f58865510b7b5522432e025c337: Status 404 returned error can't find the container with id f28f9536b15d44898b61bf31197c0b710da91f58865510b7b5522432e025c337
Oct 30 20:02:26.715467 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.715438    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:26.715556 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.715500    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:26.715556 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:26.715528    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:02:27.219266 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:27.219225    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh" podUID="c21754ed-109f-45af-ad88-de64f423c0fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:02:27.591562 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:27.591467    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" event={"ID":"442ef4a0-7909-42c2-8567-8d7555361aee","Type":"ContainerStarted","Data":"b6963a6d9c775339a268c2b7177c996fa08505f9c700ee11f4c5ad2be4090672"}
Oct 30 20:02:27.591562 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:27.591534    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" event={"ID":"442ef4a0-7909-42c2-8567-8d7555361aee","Type":"ContainerStarted","Data":"f28f9536b15d44898b61bf31197c0b710da91f58865510b7b5522432e025c337"}
Oct 30 20:02:27.708752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:27.708684    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podStartSLOduration=2.7086636520000003 podStartE2EDuration="2.708663652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:27.707628058 +0000 UTC m=+2794.918845202" watchObservedRunningTime="2026-10-30 20:02:27.708663652 +0000 UTC m=+2794.919880798"
Oct 30 20:02:28.287898 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:28.287864    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:28.290071 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:28.290042    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-748lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:02:28.290231 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:28.290103    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 20:02:28.901951 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:28.901904    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main" containerID="cri-o://6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac" gracePeriod=45
Oct 30 20:02:28.932997 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:02:28.932958    4420 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb23f3a_c315_4352_9e20_dfaa96cbe3c2.slice/crio-6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fb23f3a_c315_4352_9e20_dfaa96cbe3c2.slice/crio-conmon-6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 20:02:29.198642 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.198612    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 20:02:29.241437 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.241404    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tls-certs\") pod \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") "
Oct 30 20:02:29.241634 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.241456    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-dshm\") pod \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") "
Oct 30 20:02:29.241634 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.241500    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tmp-dir\") pod \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") "
Oct 30 20:02:29.241634 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.241528    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-model-cache\") pod \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") "
Oct 30 20:02:29.241634 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.241555    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-kube-api-access-j8kfc\" (UniqueName: \"kubernetes.io/projected/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-kube-api-access-j8kfc\") pod \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") "
Oct 30 20:02:29.241634 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.241587    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-home\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-home\") pod \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\" (UID: \"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2\") "
Oct 30 20:02:29.241939 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.241888    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tmp-dir" pod "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" (UID: "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.242031 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.242006    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-model-cache" pod "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" (UID: "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.242111 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.242090    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.242175 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.242098    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-home" pod "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" (UID: "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.244013 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.243984    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-dshm" pod "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" (UID: "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:02:29.244140 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.244015    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-kube-api-access-j8kfc" pod "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" (UID: "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2"). InnerVolumeSpecName "kube-api-access-j8kfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:29.244140 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.244077    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tls-certs" pod "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" (UID: "3fb23f3a-c315-4352-9e20-dfaa96cbe3c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:29.288355 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.288317    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-748lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:02:29.288848 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.288379    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 20:02:29.342771 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.342730    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.342771 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.342758    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.342771 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.342766    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.342771 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.342777    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8kfc\" (UniqueName: \"kubernetes.io/projected/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-kube-api-access-j8kfc\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.343142 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.342786    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:29.603293 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.603194    4420 generic.go:356] "Generic (PLEG): container finished" podID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerID="6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac" exitCode=0
Oct 30 20:02:29.603293 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.603244    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"
Oct 30 20:02:29.603508 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.603280    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" event={"ID":"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2","Type":"ContainerDied","Data":"6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac"}
Oct 30 20:02:29.603508 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.603333    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt" event={"ID":"3fb23f3a-c315-4352-9e20-dfaa96cbe3c2","Type":"ContainerDied","Data":"59c448dcbe82c68e14dfdb7879ea3ae7897f0536d1a9c33c0c727f1059bb637a"}
Oct 30 20:02:29.603508 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.603352    4420 scope.go:122] "RemoveContainer" containerID="6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac"
Oct 30 20:02:29.613610 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.613592    4420 scope.go:122] "RemoveContainer" containerID="6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac"
Oct 30 20:02:29.613944 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:02:29.613924    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac\": container with ID starting with 6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac not found: ID does not exist" containerID="6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac"
Oct 30 20:02:29.614042 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.613957    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac"} err="failed to get container status \"6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac\": rpc error: code = NotFound desc = could not find container \"6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac\": container with ID starting with 6f3d7195db2b401c7795f2348d415b23a52cde88c54229c502fa17ab81f05aac not found: ID does not exist"
Oct 30 20:02:29.778839 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.777878    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"]
Oct 30 20:02:29.778839 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:29.778447    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ada8d7/distinct-w3-ada8d7-kserve-7f97995f55-kpgwt"]
Oct 30 20:02:30.288245 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:30.288212    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-748lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:02:30.288466 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:30.288273    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 20:02:30.894972 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:30.894937    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" path="/var/lib/kubelet/pods/3fb23f3a-c315-4352-9e20-dfaa96cbe3c2/volumes"
Oct 30 20:02:31.288067 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.287978    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-748lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:02:31.288067 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.288044    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 20:02:31.430666 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.430630    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"]
Oct 30 20:02:31.431137 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.431124    4420 cpu_manager.go:414] "containerMap: removing container" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main"
Oct 30 20:02:31.431188 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.431138    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main"
Oct 30 20:02:31.431231 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.431211    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb23f3a-c315-4352-9e20-dfaa96cbe3c2" containerName="main"
Oct 30 20:02:31.434227 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.434209    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.489887 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.489691    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6dd\" (UniqueName: \"kubernetes.io/projected/cb27de52-e22a-4b73-bef1-d556a562d1e8-kube-api-access-7g6dd\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.489887 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.489775    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-model-cache\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.490363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.489901    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-tmp-dir\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.490363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.490070    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-dshm\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.490363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.490103    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-home\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.490363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.490226    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb27de52-e22a-4b73-bef1-d556a562d1e8-tls-certs\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.535425 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.535385    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f8a9d8\"/\"kube-root-ca.crt\""
Oct 30 20:02:31.535597 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.535462    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f8a9d8\"/\"openshift-service-ca.crt\""
Oct 30 20:02:31.535597 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.535493    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f8a9d8\"/\"default-dockercfg-fx628\""
Oct 30 20:02:31.538934 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.538882    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f8a9d8\"/\"test-model-f8a9d8-kserve-self-signed-certs\""
Oct 30 20:02:31.540952 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.540934    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"]
Oct 30 20:02:31.591523 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.591492    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6dd\" (UniqueName: \"kubernetes.io/projected/cb27de52-e22a-4b73-bef1-d556a562d1e8-kube-api-access-7g6dd\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.591727 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.591548    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-model-cache\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.591727 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.591581    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-tmp-dir\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.591727 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.591706    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-dshm\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.591873 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.591733    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-home\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.591873 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.591758    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb27de52-e22a-4b73-bef1-d556a562d1e8-tls-certs\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.592642 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.592057    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-model-cache\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.592642 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.592359    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-tmp-dir\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.592642 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.592598    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-home\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.594264 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.594238    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-dshm\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.594912 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.594884    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb27de52-e22a-4b73-bef1-d556a562d1e8-tls-certs\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.648359 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.648323    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6dd\" (UniqueName: \"kubernetes.io/projected/cb27de52-e22a-4b73-bef1-d556a562d1e8-kube-api-access-7g6dd\") pod \"test-model-f8a9d8-kserve-769cb8f779-zpnwl\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") " pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:31.745976 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:31.745942    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:32.144716 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:32.144694    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"]
Oct 30 20:02:32.147165 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:02:32.147136    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb27de52_e22a_4b73_bef1_d556a562d1e8.slice/crio-dbed5c727f146f023712b524d926a2be5e12098c3f67fc84b6d274575ed64007 WatchSource:0}: Error finding container dbed5c727f146f023712b524d926a2be5e12098c3f67fc84b6d274575ed64007: Status 404 returned error can't find the container with id dbed5c727f146f023712b524d926a2be5e12098c3f67fc84b6d274575ed64007
Oct 30 20:02:32.288054 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:32.288014    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-748lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:02:32.288225 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:32.288084    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 20:02:32.622099 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:32.621998    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl" event={"ID":"cb27de52-e22a-4b73-bef1-d556a562d1e8","Type":"ContainerStarted","Data":"a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1"}
Oct 30 20:02:32.622099 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:32.622051    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl" event={"ID":"cb27de52-e22a-4b73-bef1-d556a562d1e8","Type":"ContainerStarted","Data":"dbed5c727f146f023712b524d926a2be5e12098c3f67fc84b6d274575ed64007"}
Oct 30 20:02:32.736238 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:32.736159    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl" podStartSLOduration=1.7361404249999999 podStartE2EDuration="1.736140425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:32.735334253 +0000 UTC m=+2799.946551401" watchObservedRunningTime="2026-10-30 20:02:32.736140425 +0000 UTC m=+2799.947357574"
Oct 30 20:02:32.746320 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:32.746290    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:33.288249 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:33.288213    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-748lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 30 20:02:33.288424 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:33.288264    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 30 20:02:34.997937 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:34.997906    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:34.998397 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:34.998318    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:34.999236 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:34.999218    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:02:35.196330 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:02:35.196297    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:02:37.219488 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:37.219458    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh" podUID="c21754ed-109f-45af-ad88-de64f423c0fc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 20:02:41.747040 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:41.747008    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:41.758985 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:41.758962    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:42.692531 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:42.692506    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:02:43.862423 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:43.862399    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 20:02:44.021054 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.021024    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c21754ed-109f-45af-ad88-de64f423c0fc-kube-api-access-fksdf\" (UniqueName: \"kubernetes.io/projected/c21754ed-109f-45af-ad88-de64f423c0fc-kube-api-access-fksdf\") pod \"c21754ed-109f-45af-ad88-de64f423c0fc\" (UID: \"c21754ed-109f-45af-ad88-de64f423c0fc\") "
Oct 30 20:02:44.021250 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.021129    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c21754ed-109f-45af-ad88-de64f423c0fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21754ed-109f-45af-ad88-de64f423c0fc-plugins-config-volume\") pod \"c21754ed-109f-45af-ad88-de64f423c0fc\" (UID: \"c21754ed-109f-45af-ad88-de64f423c0fc\") "
Oct 30 20:02:44.021483 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.021457    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c21754ed-109f-45af-ad88-de64f423c0fc-plugins-config-volume" pod "c21754ed-109f-45af-ad88-de64f423c0fc" (UID: "c21754ed-109f-45af-ad88-de64f423c0fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:44.021699 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.021653    4420 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c21754ed-109f-45af-ad88-de64f423c0fc-plugins-config-volume\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:44.023350 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.023330    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c21754ed-109f-45af-ad88-de64f423c0fc-kube-api-access-fksdf" pod "c21754ed-109f-45af-ad88-de64f423c0fc" (UID: "c21754ed-109f-45af-ad88-de64f423c0fc"). InnerVolumeSpecName "kube-api-access-fksdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:44.123362 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.123333    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fksdf\" (UniqueName: \"kubernetes.io/projected/c21754ed-109f-45af-ad88-de64f423c0fc-kube-api-access-fksdf\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:44.478496 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.478467    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"]
Oct 30 20:02:44.478664 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.478647    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9" podUID="a210fc35-eee1-44f1-9dc3-eb66dc04f288" containerName="maas-api" containerID="cri-o://b4d215fbc22e1cb8094ec94f2c62bfa9f2573561ba3f5355a9b5d50535ffc073" gracePeriod=30
Oct 30 20:02:44.695554 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.695518    4420 generic.go:356] "Generic (PLEG): container finished" podID="c21754ed-109f-45af-ad88-de64f423c0fc" containerID="5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9" exitCode=0
Oct 30 20:02:44.695722 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.695583    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"
Oct 30 20:02:44.695722 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.695637    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh" event={"ID":"c21754ed-109f-45af-ad88-de64f423c0fc","Type":"ContainerDied","Data":"5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9"}
Oct 30 20:02:44.695722 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.695690    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh" event={"ID":"c21754ed-109f-45af-ad88-de64f423c0fc","Type":"ContainerDied","Data":"8d910200047c204db1edef2f69f789d63f4d74b2e6a5477b6202582ec5a12bde"}
Oct 30 20:02:44.695722 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.695714    4420 scope.go:122] "RemoveContainer" containerID="5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9"
Oct 30 20:02:44.697607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.697585    4420 generic.go:356] "Generic (PLEG): container finished" podID="a210fc35-eee1-44f1-9dc3-eb66dc04f288" containerID="b4d215fbc22e1cb8094ec94f2c62bfa9f2573561ba3f5355a9b5d50535ffc073" exitCode=0
Oct 30 20:02:44.697709 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.697638    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9" event={"ID":"a210fc35-eee1-44f1-9dc3-eb66dc04f288","Type":"ContainerDied","Data":"b4d215fbc22e1cb8094ec94f2c62bfa9f2573561ba3f5355a9b5d50535ffc073"}
Oct 30 20:02:44.706242 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.706225    4420 scope.go:122] "RemoveContainer" containerID="5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9"
Oct 30 20:02:44.706508 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:02:44.706485    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9\": container with ID starting with 5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9 not found: ID does not exist" containerID="5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9"
Oct 30 20:02:44.706568 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.706528    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9"} err="failed to get container status \"5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9\": rpc error: code = NotFound desc = could not find container \"5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9\": container with ID starting with 5b9e2e9ab76e6af02d25ac3c521fab6dee3866646f6aca08590b9698a3e12ee9 not found: ID does not exist"
Oct 30 20:02:44.742996 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.742946    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 20:02:44.936674 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.936641    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-maas-api-tls\") pod \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") "
Oct 30 20:02:44.937090 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.936722    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-metrics-certs\") pod \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") "
Oct 30 20:02:44.937090 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.936840    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a210fc35-eee1-44f1-9dc3-eb66dc04f288-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a210fc35-eee1-44f1-9dc3-eb66dc04f288-openshift-service-ca\") pod \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") "
Oct 30 20:02:44.937090 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.936924    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a210fc35-eee1-44f1-9dc3-eb66dc04f288-kube-api-access-n2wjp\" (UniqueName: \"kubernetes.io/projected/a210fc35-eee1-44f1-9dc3-eb66dc04f288-kube-api-access-n2wjp\") pod \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\" (UID: \"a210fc35-eee1-44f1-9dc3-eb66dc04f288\") "
Oct 30 20:02:44.937235 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.937168    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a210fc35-eee1-44f1-9dc3-eb66dc04f288-openshift-service-ca" pod "a210fc35-eee1-44f1-9dc3-eb66dc04f288" (UID: "a210fc35-eee1-44f1-9dc3-eb66dc04f288"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:44.937466 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.937406    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a210fc35-eee1-44f1-9dc3-eb66dc04f288-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:44.939082 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.939061    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-metrics-certs" pod "a210fc35-eee1-44f1-9dc3-eb66dc04f288" (UID: "a210fc35-eee1-44f1-9dc3-eb66dc04f288"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:44.939181 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.939121    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a210fc35-eee1-44f1-9dc3-eb66dc04f288-kube-api-access-n2wjp" pod "a210fc35-eee1-44f1-9dc3-eb66dc04f288" (UID: "a210fc35-eee1-44f1-9dc3-eb66dc04f288"). InnerVolumeSpecName "kube-api-access-n2wjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:44.939357 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.939340    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-maas-api-tls" pod "a210fc35-eee1-44f1-9dc3-eb66dc04f288" (UID: "a210fc35-eee1-44f1-9dc3-eb66dc04f288"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:44.989652 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.989616    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"]
Oct 30 20:02:44.993057 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:44.992972    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a3f9b7-6d49bc7b67-ct9nh"]
Oct 30 20:02:45.038831 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:45.038783    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.038831 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:45.038806    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a210fc35-eee1-44f1-9dc3-eb66dc04f288-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.038831 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:45.038833    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2wjp\" (UniqueName: \"kubernetes.io/projected/a210fc35-eee1-44f1-9dc3-eb66dc04f288-kube-api-access-n2wjp\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:45.602883 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:45.602852    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"]
Oct 30 20:02:45.603091 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:45.603052    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94" podUID="2b8980fb-7181-4478-a3f1-89d285253ff7" containerName="maas-api" containerID="cri-o://be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1" gracePeriod=30
Oct 30 20:02:45.705620 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:45.705587    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9" event={"ID":"a210fc35-eee1-44f1-9dc3-eb66dc04f288","Type":"ContainerDied","Data":"f353e601c66de306066a8c97597cd7dd533034df2c8fba6211f47a9797d5dbee"}
Oct 30 20:02:45.705620 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:45.705615    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"
Oct 30 20:02:45.705864 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:45.705644    4420 scope.go:122] "RemoveContainer" containerID="b4d215fbc22e1cb8094ec94f2c62bfa9f2573561ba3f5355a9b5d50535ffc073"
Oct 30 20:02:45.829787 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:45.829752    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"]
Oct 30 20:02:45.832050 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:45.832024    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ada8d7-7747586c6b-24mk9"]
Oct 30 20:02:45.858964 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:45.858907    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 20:02:46.048827 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.048797    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8980fb-7181-4478-a3f1-89d285253ff7-kube-api-access-ptm5q\" (UniqueName: \"kubernetes.io/projected/2b8980fb-7181-4478-a3f1-89d285253ff7-kube-api-access-ptm5q\") pod \"2b8980fb-7181-4478-a3f1-89d285253ff7\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") "
Oct 30 20:02:46.049245 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.048915    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-metrics-certs\") pod \"2b8980fb-7181-4478-a3f1-89d285253ff7\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") "
Oct 30 20:02:46.049245 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.048956    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-maas-api-tls\") pod \"2b8980fb-7181-4478-a3f1-89d285253ff7\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") "
Oct 30 20:02:46.049245 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.049032    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b8980fb-7181-4478-a3f1-89d285253ff7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8980fb-7181-4478-a3f1-89d285253ff7-openshift-service-ca\") pod \"2b8980fb-7181-4478-a3f1-89d285253ff7\" (UID: \"2b8980fb-7181-4478-a3f1-89d285253ff7\") "
Oct 30 20:02:46.049470 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.049432    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8980fb-7181-4478-a3f1-89d285253ff7-openshift-service-ca" pod "2b8980fb-7181-4478-a3f1-89d285253ff7" (UID: "2b8980fb-7181-4478-a3f1-89d285253ff7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:02:46.051182 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.051154    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-metrics-certs" pod "2b8980fb-7181-4478-a3f1-89d285253ff7" (UID: "2b8980fb-7181-4478-a3f1-89d285253ff7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:46.051272 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.051202    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8980fb-7181-4478-a3f1-89d285253ff7-kube-api-access-ptm5q" pod "2b8980fb-7181-4478-a3f1-89d285253ff7" (UID: "2b8980fb-7181-4478-a3f1-89d285253ff7"). InnerVolumeSpecName "kube-api-access-ptm5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:02:46.051390 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.051375    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-maas-api-tls" pod "2b8980fb-7181-4478-a3f1-89d285253ff7" (UID: "2b8980fb-7181-4478-a3f1-89d285253ff7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:02:46.150272 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.150246    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptm5q\" (UniqueName: \"kubernetes.io/projected/2b8980fb-7181-4478-a3f1-89d285253ff7-kube-api-access-ptm5q\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:46.150272 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.150271    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:46.150482 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.150283    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b8980fb-7181-4478-a3f1-89d285253ff7-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:46.150482 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.150292    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b8980fb-7181-4478-a3f1-89d285253ff7-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:02:46.713559 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.713519    4420 generic.go:356] "Generic (PLEG): container finished" podID="2b8980fb-7181-4478-a3f1-89d285253ff7" containerID="be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1" exitCode=0
Oct 30 20:02:46.713737 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.713569    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"
Oct 30 20:02:46.713737 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.713602    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94" event={"ID":"2b8980fb-7181-4478-a3f1-89d285253ff7","Type":"ContainerDied","Data":"be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1"}
Oct 30 20:02:46.713737 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.713663    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94" event={"ID":"2b8980fb-7181-4478-a3f1-89d285253ff7","Type":"ContainerDied","Data":"759e2afb04dae7dd7ada3b98c8240231b35380c64d46d26deb40bed1f2ccdd66"}
Oct 30 20:02:46.713737 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.713687    4420 scope.go:122] "RemoveContainer" containerID="be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1"
Oct 30 20:02:46.724025 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.724006    4420 scope.go:122] "RemoveContainer" containerID="be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1"
Oct 30 20:02:46.724285 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:02:46.724264    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1\": container with ID starting with be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1 not found: ID does not exist" containerID="be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1"
Oct 30 20:02:46.724341 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.724293    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1"} err="failed to get container status \"be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1\": rpc error: code = NotFound desc = could not find container \"be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1\": container with ID starting with be08e13362251fe64d8c776b8b057df4bcbf62d17d64f9bec73fd979576328d1 not found: ID does not exist"
Oct 30 20:02:46.877525 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.877494    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"]
Oct 30 20:02:46.881401 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.881376    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-a3f9b7-5dc6b5d4f6-27j94"]
Oct 30 20:02:46.894903 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.894873    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8980fb-7181-4478-a3f1-89d285253ff7" path="/var/lib/kubelet/pods/2b8980fb-7181-4478-a3f1-89d285253ff7/volumes"
Oct 30 20:02:46.895336 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.895321    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a210fc35-eee1-44f1-9dc3-eb66dc04f288" path="/var/lib/kubelet/pods/a210fc35-eee1-44f1-9dc3-eb66dc04f288/volumes"
Oct 30 20:02:46.895685 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:02:46.895672    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c21754ed-109f-45af-ad88-de64f423c0fc" path="/var/lib/kubelet/pods/c21754ed-109f-45af-ad88-de64f423c0fc/volumes"
Oct 30 20:03:05.195378 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:03:05.195293    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:03:35.196415 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:03:35.196382    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:04:05.196708 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:04:05.196668    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:04:20.613055 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:20.613020    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"]
Oct 30 20:04:35.194933 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:04:35.194849    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:04:35.634200 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.634085    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg" podUID="a52f1425-a774-4d44-aa7c-1b0ddb750fb9" containerName="main" containerID="cri-o://165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19" gracePeriod=45
Oct 30 20:04:35.901126 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.901096    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:04:35.974035 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.974011    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-home\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-home\") pod \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") "
Oct 30 20:04:35.974192 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.974052    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-model-cache\") pod \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") "
Oct 30 20:04:35.974192 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.974071    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tmp-dir\") pod \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") "
Oct 30 20:04:35.974192 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.974139    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-dshm\") pod \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") "
Oct 30 20:04:35.974192 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.974184    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-kube-api-access-kd7bp\" (UniqueName: \"kubernetes.io/projected/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-kube-api-access-kd7bp\") pod \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") "
Oct 30 20:04:35.974422 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.974251    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tls-certs\" (UniqueName: \"kubernetes.io/secret/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tls-certs\") pod \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\" (UID: \"a52f1425-a774-4d44-aa7c-1b0ddb750fb9\") "
Oct 30 20:04:35.974422 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.974315    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-model-cache" pod "a52f1425-a774-4d44-aa7c-1b0ddb750fb9" (UID: "a52f1425-a774-4d44-aa7c-1b0ddb750fb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:35.974422 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.974330    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-home" pod "a52f1425-a774-4d44-aa7c-1b0ddb750fb9" (UID: "a52f1425-a774-4d44-aa7c-1b0ddb750fb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:35.974683 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.974630    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:35.974683 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.974660    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:35.974911 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.974678    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tmp-dir" pod "a52f1425-a774-4d44-aa7c-1b0ddb750fb9" (UID: "a52f1425-a774-4d44-aa7c-1b0ddb750fb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:35.976752 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.976718    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-dshm" pod "a52f1425-a774-4d44-aa7c-1b0ddb750fb9" (UID: "a52f1425-a774-4d44-aa7c-1b0ddb750fb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:04:35.977211 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.977188    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-kube-api-access-kd7bp" pod "a52f1425-a774-4d44-aa7c-1b0ddb750fb9" (UID: "a52f1425-a774-4d44-aa7c-1b0ddb750fb9"). InnerVolumeSpecName "kube-api-access-kd7bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:04:35.980926 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:35.980901    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tls-certs" pod "a52f1425-a774-4d44-aa7c-1b0ddb750fb9" (UID: "a52f1425-a774-4d44-aa7c-1b0ddb750fb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:04:36.075913 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.075889    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:36.075913 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.075913    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:36.076071 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.075923    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd7bp\" (UniqueName: \"kubernetes.io/projected/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-kube-api-access-kd7bp\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:36.076071 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.075931    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52f1425-a774-4d44-aa7c-1b0ddb750fb9-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:04:36.258550 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.258471    4420 generic.go:356] "Generic (PLEG): container finished" podID="a52f1425-a774-4d44-aa7c-1b0ddb750fb9" containerID="165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19" exitCode=0
Oct 30 20:04:36.258550 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.258519    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"
Oct 30 20:04:36.259060 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.258560    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg" event={"ID":"a52f1425-a774-4d44-aa7c-1b0ddb750fb9","Type":"ContainerDied","Data":"165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19"}
Oct 30 20:04:36.259060 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.258600    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg" event={"ID":"a52f1425-a774-4d44-aa7c-1b0ddb750fb9","Type":"ContainerDied","Data":"99c6eeb6d81c6d447a896a19aab9c20d953da1210126b83c3ab2485045d223dd"}
Oct 30 20:04:36.259060 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.258617    4420 scope.go:122] "RemoveContainer" containerID="165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19"
Oct 30 20:04:36.268394 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.268373    4420 scope.go:122] "RemoveContainer" containerID="165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19"
Oct 30 20:04:36.268693 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:04:36.268674    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19\": container with ID starting with 165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19 not found: ID does not exist" containerID="165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19"
Oct 30 20:04:36.268763 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.268700    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19"} err="failed to get container status \"165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19\": rpc error: code = NotFound desc = could not find container \"165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19\": container with ID starting with 165f1d05c463058d625a5286239cbad59a8115f9f802acd3d02e6ccd34ceeb19 not found: ID does not exist"
Oct 30 20:04:36.425086 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.425063    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"]
Oct 30 20:04:36.429422 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.429400    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-132119/test-model-132119-kserve-6b5d445fdf-6x7jg"]
Oct 30 20:04:36.896303 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:36.896272    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a52f1425-a774-4d44-aa7c-1b0ddb750fb9" path="/var/lib/kubelet/pods/a52f1425-a774-4d44-aa7c-1b0ddb750fb9/volumes"
Oct 30 20:04:46.710984 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:04:46.710946    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"]
Oct 30 20:05:01.730552 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:01.730500    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl" podUID="cb27de52-e22a-4b73-bef1-d556a562d1e8" containerName="main" containerID="cri-o://a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1" gracePeriod=45
Oct 30 20:05:01.979014 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:01.978992    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:05:02.005667 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.005607    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-dshm\") pod \"cb27de52-e22a-4b73-bef1-d556a562d1e8\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") "
Oct 30 20:05:02.005800 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.005697    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-home\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-home\") pod \"cb27de52-e22a-4b73-bef1-d556a562d1e8\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") "
Oct 30 20:05:02.005800 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.005737    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-model-cache\") pod \"cb27de52-e22a-4b73-bef1-d556a562d1e8\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") "
Oct 30 20:05:02.005800 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.005771    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-tmp-dir\") pod \"cb27de52-e22a-4b73-bef1-d556a562d1e8\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") "
Oct 30 20:05:02.006045 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.005841    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb27de52-e22a-4b73-bef1-d556a562d1e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb27de52-e22a-4b73-bef1-d556a562d1e8-tls-certs\") pod \"cb27de52-e22a-4b73-bef1-d556a562d1e8\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") "
Oct 30 20:05:02.006045 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.005966    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb27de52-e22a-4b73-bef1-d556a562d1e8-kube-api-access-7g6dd\" (UniqueName: \"kubernetes.io/projected/cb27de52-e22a-4b73-bef1-d556a562d1e8-kube-api-access-7g6dd\") pod \"cb27de52-e22a-4b73-bef1-d556a562d1e8\" (UID: \"cb27de52-e22a-4b73-bef1-d556a562d1e8\") "
Oct 30 20:05:02.006045 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.005990    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-model-cache" pod "cb27de52-e22a-4b73-bef1-d556a562d1e8" (UID: "cb27de52-e22a-4b73-bef1-d556a562d1e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:02.006045 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.005975    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-home" pod "cb27de52-e22a-4b73-bef1-d556a562d1e8" (UID: "cb27de52-e22a-4b73-bef1-d556a562d1e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:02.006045 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.006032    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-tmp-dir" pod "cb27de52-e22a-4b73-bef1-d556a562d1e8" (UID: "cb27de52-e22a-4b73-bef1-d556a562d1e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:02.006384 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.006367    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.006454 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.006389    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.006454 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.006402    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.008006 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.007981    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb27de52-e22a-4b73-bef1-d556a562d1e8-tls-certs" pod "cb27de52-e22a-4b73-bef1-d556a562d1e8" (UID: "cb27de52-e22a-4b73-bef1-d556a562d1e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:02.008125 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.008049    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-dshm" pod "cb27de52-e22a-4b73-bef1-d556a562d1e8" (UID: "cb27de52-e22a-4b73-bef1-d556a562d1e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:05:02.008125 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.008070    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb27de52-e22a-4b73-bef1-d556a562d1e8-kube-api-access-7g6dd" pod "cb27de52-e22a-4b73-bef1-d556a562d1e8" (UID: "cb27de52-e22a-4b73-bef1-d556a562d1e8"). InnerVolumeSpecName "kube-api-access-7g6dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:02.107890 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.107863    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb27de52-e22a-4b73-bef1-d556a562d1e8-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.107890 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.107890    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb27de52-e22a-4b73-bef1-d556a562d1e8-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.108060 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.107902    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g6dd\" (UniqueName: \"kubernetes.io/projected/cb27de52-e22a-4b73-bef1-d556a562d1e8-kube-api-access-7g6dd\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:02.384206 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.384126    4420 generic.go:356] "Generic (PLEG): container finished" podID="cb27de52-e22a-4b73-bef1-d556a562d1e8" containerID="a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1" exitCode=0
Oct 30 20:05:02.384206 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.384168    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"
Oct 30 20:05:02.384439 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.384213    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl" event={"ID":"cb27de52-e22a-4b73-bef1-d556a562d1e8","Type":"ContainerDied","Data":"a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1"}
Oct 30 20:05:02.384439 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.384249    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl" event={"ID":"cb27de52-e22a-4b73-bef1-d556a562d1e8","Type":"ContainerDied","Data":"dbed5c727f146f023712b524d926a2be5e12098c3f67fc84b6d274575ed64007"}
Oct 30 20:05:02.384439 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.384270    4420 scope.go:122] "RemoveContainer" containerID="a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1"
Oct 30 20:05:02.394163 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.394142    4420 scope.go:122] "RemoveContainer" containerID="a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1"
Oct 30 20:05:02.394445 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:05:02.394429    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1\": container with ID starting with a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1 not found: ID does not exist" containerID="a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1"
Oct 30 20:05:02.394505 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.394458    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1"} err="failed to get container status \"a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1\": rpc error: code = NotFound desc = could not find container \"a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1\": container with ID starting with a0cdfa63be9360b71d75a8c7215ecab42c86b81f1a1019cdf3601ad382b18ef1 not found: ID does not exist"
Oct 30 20:05:02.545999 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.545980    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"]
Oct 30 20:05:02.595041 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.595015    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f8a9d8/test-model-f8a9d8-kserve-769cb8f779-zpnwl"]
Oct 30 20:05:02.894916 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:02.894890    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb27de52-e22a-4b73-bef1-d556a562d1e8" path="/var/lib/kubelet/pods/cb27de52-e22a-4b73-bef1-d556a562d1e8/volumes"
Oct 30 20:05:05.195044 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:05:05.195010    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:05:12.512100 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.512068    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"]
Oct 30 20:05:12.512649 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.512225    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq" podUID="0a8fddd9-8d89-4434-9bc1-7dd44e8f959d" containerName="maas-api" containerID="cri-o://69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e" gracePeriod=30
Oct 30 20:05:12.763203 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.763147    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 20:05:12.795582 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.795551    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-maas-api-tls\") pod \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") "
Oct 30 20:05:12.795727 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.795608    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-kube-api-access-5pg7f\" (UniqueName: \"kubernetes.io/projected/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-kube-api-access-5pg7f\") pod \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") "
Oct 30 20:05:12.795727 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.795669    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-openshift-service-ca\") pod \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") "
Oct 30 20:05:12.795886 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.795784    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-metrics-certs\") pod \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\" (UID: \"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d\") "
Oct 30 20:05:12.796069 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.796044    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-openshift-service-ca" pod "0a8fddd9-8d89-4434-9bc1-7dd44e8f959d" (UID: "0a8fddd9-8d89-4434-9bc1-7dd44e8f959d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:12.796189 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.796172    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:12.797781 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.797760    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-maas-api-tls" pod "0a8fddd9-8d89-4434-9bc1-7dd44e8f959d" (UID: "0a8fddd9-8d89-4434-9bc1-7dd44e8f959d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:12.797885 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.797777    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-metrics-certs" pod "0a8fddd9-8d89-4434-9bc1-7dd44e8f959d" (UID: "0a8fddd9-8d89-4434-9bc1-7dd44e8f959d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:12.797946 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.797903    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-kube-api-access-5pg7f" pod "0a8fddd9-8d89-4434-9bc1-7dd44e8f959d" (UID: "0a8fddd9-8d89-4434-9bc1-7dd44e8f959d"). InnerVolumeSpecName "kube-api-access-5pg7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:12.896645 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.896621    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:12.896645 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.896644    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pg7f\" (UniqueName: \"kubernetes.io/projected/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-kube-api-access-5pg7f\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:12.896906 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:12.896654    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:13.448097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:13.445802    4420 generic.go:356] "Generic (PLEG): container finished" podID="0a8fddd9-8d89-4434-9bc1-7dd44e8f959d" containerID="69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e" exitCode=0
Oct 30 20:05:13.448097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:13.445985    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq" event={"ID":"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d","Type":"ContainerDied","Data":"69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e"}
Oct 30 20:05:13.448097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:13.446024    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq" event={"ID":"0a8fddd9-8d89-4434-9bc1-7dd44e8f959d","Type":"ContainerDied","Data":"498c4f0ca373830bf6e6836a7fa46497602b1a83d4702eac6373a6ea86885bb1"}
Oct 30 20:05:13.448097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:13.446051    4420 scope.go:122] "RemoveContainer" containerID="69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e"
Oct 30 20:05:13.448097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:13.446181    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"
Oct 30 20:05:13.460510 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:13.460485    4420 scope.go:122] "RemoveContainer" containerID="69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e"
Oct 30 20:05:13.460887 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:05:13.460860    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e\": container with ID starting with 69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e not found: ID does not exist" containerID="69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e"
Oct 30 20:05:13.460972 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:13.460904    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e"} err="failed to get container status \"69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e\": rpc error: code = NotFound desc = could not find container \"69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e\": container with ID starting with 69e27776921db04d34eb1fafd87690b36b112a8db09d174f0d17921e61447c5e not found: ID does not exist"
Oct 30 20:05:13.607842 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:13.607804    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"]
Oct 30 20:05:13.614160 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:13.614133    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-132119-7dc5f8bf86-dtvmq"]
Oct 30 20:05:14.900940 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:14.900900    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8fddd9-8d89-4434-9bc1-7dd44e8f959d" path="/var/lib/kubelet/pods/0a8fddd9-8d89-4434-9bc1-7dd44e8f959d/volumes"
Oct 30 20:05:35.196004 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:05:35.195969    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:05:46.950620 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:46.950588    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"]
Oct 30 20:05:46.951168 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:46.950772    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc" podUID="21ece487-7e77-4656-8f3c-ea9131e437bf" containerName="maas-api" containerID="cri-o://8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b" gracePeriod=30
Oct 30 20:05:47.213279 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.213218    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:05:47.290387 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.290359    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-metrics-certs\") pod \"21ece487-7e77-4656-8f3c-ea9131e437bf\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") "
Oct 30 20:05:47.290541 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.290455    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-maas-api-tls\") pod \"21ece487-7e77-4656-8f3c-ea9131e437bf\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") "
Oct 30 20:05:47.290541 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.290486    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ece487-7e77-4656-8f3c-ea9131e437bf-kube-api-access-wqqq8\" (UniqueName: \"kubernetes.io/projected/21ece487-7e77-4656-8f3c-ea9131e437bf-kube-api-access-wqqq8\") pod \"21ece487-7e77-4656-8f3c-ea9131e437bf\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") "
Oct 30 20:05:47.290541 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.290515    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ece487-7e77-4656-8f3c-ea9131e437bf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ece487-7e77-4656-8f3c-ea9131e437bf-openshift-service-ca\") pod \"21ece487-7e77-4656-8f3c-ea9131e437bf\" (UID: \"21ece487-7e77-4656-8f3c-ea9131e437bf\") "
Oct 30 20:05:47.290947 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.290916    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ece487-7e77-4656-8f3c-ea9131e437bf-openshift-service-ca" pod "21ece487-7e77-4656-8f3c-ea9131e437bf" (UID: "21ece487-7e77-4656-8f3c-ea9131e437bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:05:47.292607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.292582    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-metrics-certs" pod "21ece487-7e77-4656-8f3c-ea9131e437bf" (UID: "21ece487-7e77-4656-8f3c-ea9131e437bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:47.292693 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.292636    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ece487-7e77-4656-8f3c-ea9131e437bf-kube-api-access-wqqq8" pod "21ece487-7e77-4656-8f3c-ea9131e437bf" (UID: "21ece487-7e77-4656-8f3c-ea9131e437bf"). InnerVolumeSpecName "kube-api-access-wqqq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:05:47.292885 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.292860    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-maas-api-tls" pod "21ece487-7e77-4656-8f3c-ea9131e437bf" (UID: "21ece487-7e77-4656-8f3c-ea9131e437bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:05:47.392323 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.392294    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:47.392323 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.392322    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqqq8\" (UniqueName: \"kubernetes.io/projected/21ece487-7e77-4656-8f3c-ea9131e437bf-kube-api-access-wqqq8\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:47.392465 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.392336    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21ece487-7e77-4656-8f3c-ea9131e437bf-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:47.392465 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.392349    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21ece487-7e77-4656-8f3c-ea9131e437bf-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:05:47.608392 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.608311    4420 generic.go:356] "Generic (PLEG): container finished" podID="21ece487-7e77-4656-8f3c-ea9131e437bf" containerID="8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b" exitCode=0
Oct 30 20:05:47.608392 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.608368    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"
Oct 30 20:05:47.608592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.608395    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc" event={"ID":"21ece487-7e77-4656-8f3c-ea9131e437bf","Type":"ContainerDied","Data":"8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b"}
Oct 30 20:05:47.608592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.608476    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc" event={"ID":"21ece487-7e77-4656-8f3c-ea9131e437bf","Type":"ContainerDied","Data":"a90e66f8a7ed93bd8f741dff55d5b37ececf0535ba0bd18f8314d5d416250b75"}
Oct 30 20:05:47.608592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.608498    4420 scope.go:122] "RemoveContainer" containerID="8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b"
Oct 30 20:05:47.617661 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.617642    4420 scope.go:122] "RemoveContainer" containerID="8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b"
Oct 30 20:05:47.617997 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:05:47.617979    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b\": container with ID starting with 8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b not found: ID does not exist" containerID="8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b"
Oct 30 20:05:47.618077 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.618008    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b"} err="failed to get container status \"8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b\": rpc error: code = NotFound desc = could not find container \"8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b\": container with ID starting with 8d6a7bf77f3d888276aa82af91a9ddaa8212b53185216c0eb991ca2dbe15457b not found: ID does not exist"
Oct 30 20:05:47.770239 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.770208    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"]
Oct 30 20:05:47.774637 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:47.774612    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8a9d8-78ff98d758-dzrfc"]
Oct 30 20:05:48.894612 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:48.894578    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ece487-7e77-4656-8f3c-ea9131e437bf" path="/var/lib/kubelet/pods/21ece487-7e77-4656-8f3c-ea9131e437bf/volumes"
Oct 30 20:05:56.370854 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:56.370807    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"]
Oct 30 20:05:56.371272 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:56.371025    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy" containerID="cri-o://3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957" gracePeriod=30
Oct 30 20:05:57.111463 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111429    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"]
Oct 30 20:05:57.111834 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111807    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a210fc35-eee1-44f1-9dc3-eb66dc04f288" containerName="maas-api"
Oct 30 20:05:57.111894 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111834    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a210fc35-eee1-44f1-9dc3-eb66dc04f288" containerName="maas-api"
Oct 30 20:05:57.111894 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111856    4420 cpu_manager.go:414] "containerMap: removing container" podUID="21ece487-7e77-4656-8f3c-ea9131e437bf" containerName="maas-api"
Oct 30 20:05:57.111894 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111860    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ece487-7e77-4656-8f3c-ea9131e437bf" containerName="maas-api"
Oct 30 20:05:57.111894 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111871    4420 cpu_manager.go:414] "containerMap: removing container" podUID="2b8980fb-7181-4478-a3f1-89d285253ff7" containerName="maas-api"
Oct 30 20:05:57.111894 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111876    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8980fb-7181-4478-a3f1-89d285253ff7" containerName="maas-api"
Oct 30 20:05:57.111894 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111883    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a52f1425-a774-4d44-aa7c-1b0ddb750fb9" containerName="main"
Oct 30 20:05:57.111894 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111887    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a52f1425-a774-4d44-aa7c-1b0ddb750fb9" containerName="main"
Oct 30 20:05:57.111894 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111896    4420 cpu_manager.go:414] "containerMap: removing container" podUID="0a8fddd9-8d89-4434-9bc1-7dd44e8f959d" containerName="maas-api"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111901    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8fddd9-8d89-4434-9bc1-7dd44e8f959d" containerName="maas-api"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111919    4420 cpu_manager.go:414] "containerMap: removing container" podUID="c21754ed-109f-45af-ad88-de64f423c0fc" containerName="payload-pre-processing"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111923    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c21754ed-109f-45af-ad88-de64f423c0fc" containerName="payload-pre-processing"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111935    4420 cpu_manager.go:414] "containerMap: removing container" podUID="cb27de52-e22a-4b73-bef1-d556a562d1e8" containerName="main"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.111939    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb27de52-e22a-4b73-bef1-d556a562d1e8" containerName="main"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.112012    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a210fc35-eee1-44f1-9dc3-eb66dc04f288" containerName="maas-api"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.112022    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ece487-7e77-4656-8f3c-ea9131e437bf" containerName="maas-api"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.112028    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8980fb-7181-4478-a3f1-89d285253ff7" containerName="maas-api"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.112034    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a52f1425-a774-4d44-aa7c-1b0ddb750fb9" containerName="main"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.112039    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8fddd9-8d89-4434-9bc1-7dd44e8f959d" containerName="maas-api"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.112046    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c21754ed-109f-45af-ad88-de64f423c0fc" containerName="payload-pre-processing"
Oct 30 20:05:57.112131 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.112052    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb27de52-e22a-4b73-bef1-d556a562d1e8" containerName="main"
Oct 30 20:05:57.116272 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.116256    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:05:57.170096 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.170073    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:05:57.219769 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.219745    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:05:57.221941 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.221922    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"]
Oct 30 20:05:57.285236 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.285207    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:05:57.285368 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.285269    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfsbc\" (UniqueName: \"kubernetes.io/projected/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-kube-api-access-pfsbc\") pod \"maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:05:57.285434 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.285371    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:05:57.386140 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.386085    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:05:57.386546 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.386169    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:05:57.386546 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.386203    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfsbc\" (UniqueName: \"kubernetes.io/projected/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-kube-api-access-pfsbc\") pod \"maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:05:57.386858 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.386841    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:05:57.388567 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.388547    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:05:57.441986 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.441958    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfsbc\" (UniqueName: \"kubernetes.io/projected/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-kube-api-access-pfsbc\") pod \"maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:05:57.728314 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:57.728287    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:05:58.100633 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:58.100609    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"]
Oct 30 20:05:58.102891 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:05:58.102865    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd26faa_70f1_4b7f_9f3f_a632fdc0508c.slice/crio-4567bdf74a04f2a7ed4dee8e502eee68609747e2ccd57fd5f014f45a16934021 WatchSource:0}: Error finding container 4567bdf74a04f2a7ed4dee8e502eee68609747e2ccd57fd5f014f45a16934021: Status 404 returned error can't find the container with id 4567bdf74a04f2a7ed4dee8e502eee68609747e2ccd57fd5f014f45a16934021
Oct 30 20:05:58.661391 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:58.661354    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx" event={"ID":"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c","Type":"ContainerStarted","Data":"8fa069e264f7946f97308e2028df7f9a66fadc35bb422595a6b98320a06aede5"}
Oct 30 20:05:58.661757 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:58.661440    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx" event={"ID":"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c","Type":"ContainerStarted","Data":"4567bdf74a04f2a7ed4dee8e502eee68609747e2ccd57fd5f014f45a16934021"}
Oct 30 20:05:58.775078 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:58.775014    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx" podStartSLOduration=1.774999125 podStartE2EDuration="1.774999125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:58.773721359 +0000 UTC m=+3005.984938506" watchObservedRunningTime="2026-10-30 20:05:58.774999125 +0000 UTC m=+3005.986216270"
Oct 30 20:05:59.667462 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:59.667432    4420 generic.go:356] "Generic (PLEG): container finished" podID="bcd26faa-70f1-4b7f-9f3f-a632fdc0508c" containerID="8fa069e264f7946f97308e2028df7f9a66fadc35bb422595a6b98320a06aede5" exitCode=0
Oct 30 20:05:59.667889 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:05:59.667518    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx" event={"ID":"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c","Type":"ContainerDied","Data":"8fa069e264f7946f97308e2028df7f9a66fadc35bb422595a6b98320a06aede5"}
Oct 30 20:06:00.807485 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:00.807464    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:06:00.814589 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:00.814562    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-xks-ca\") pod \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") "
Oct 30 20:06:00.814695 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:00.814595    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-openshift-ca\") pod \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") "
Oct 30 20:06:00.814695 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:00.814690    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-kube-api-access-pfsbc\" (UniqueName: \"kubernetes.io/projected/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-kube-api-access-pfsbc\") pod \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\" (UID: \"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c\") "
Oct 30 20:06:00.815029 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:00.815003    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-openshift-ca" pod "bcd26faa-70f1-4b7f-9f3f-a632fdc0508c" (UID: "bcd26faa-70f1-4b7f-9f3f-a632fdc0508c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:00.815136 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:00.815109    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:00.816666 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:00.816634    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-xks-ca" pod "bcd26faa-70f1-4b7f-9f3f-a632fdc0508c" (UID: "bcd26faa-70f1-4b7f-9f3f-a632fdc0508c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:00.816797 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:00.816774    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-kube-api-access-pfsbc" pod "bcd26faa-70f1-4b7f-9f3f-a632fdc0508c" (UID: "bcd26faa-70f1-4b7f-9f3f-a632fdc0508c"). InnerVolumeSpecName "kube-api-access-pfsbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:00.916261 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:00.916240    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfsbc\" (UniqueName: \"kubernetes.io/projected/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-kube-api-access-pfsbc\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:00.916261 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:00.916260    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:01.605448 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.605425    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:06:01.622562 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.622539    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-envoy\") pod \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") "
Oct 30 20:06:01.622674 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.622571    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-credential-socket\") pod \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") "
Oct 30 20:06:01.622674 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.622598    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-socket\") pod \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") "
Oct 30 20:06:01.622674 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.622623    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-data\") pod \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") "
Oct 30 20:06:01.622864 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.622685    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-podinfo\") pod \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") "
Oct 30 20:06:01.622864 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.622739    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-kube-api-access-m87l5\" (UniqueName: \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-kube-api-access-m87l5\") pod \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") "
Oct 30 20:06:01.622864 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.622780    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istiod-ca-cert\") pod \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") "
Oct 30 20:06:01.622864 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.622797    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-credential-socket" pod "21239ee1-9c9e-44e2-ab73-4ab81c3def78" (UID: "21239ee1-9c9e-44e2-ab73-4ab81c3def78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:01.622864 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.622808    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-token\" (UniqueName: \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-token\") pod \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") "
Oct 30 20:06:01.623145 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.622919    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-socket" pod "21239ee1-9c9e-44e2-ab73-4ab81c3def78" (UID: "21239ee1-9c9e-44e2-ab73-4ab81c3def78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:01.623145 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.622937    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-certs\") pod \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\" (UID: \"21239ee1-9c9e-44e2-ab73-4ab81c3def78\") "
Oct 30 20:06:01.623145 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.623006    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-data" pod "21239ee1-9c9e-44e2-ab73-4ab81c3def78" (UID: "21239ee1-9c9e-44e2-ab73-4ab81c3def78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:01.623471 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.623283    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-certs" pod "21239ee1-9c9e-44e2-ab73-4ab81c3def78" (UID: "21239ee1-9c9e-44e2-ab73-4ab81c3def78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:01.623471 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.623321    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istiod-ca-cert" pod "21239ee1-9c9e-44e2-ab73-4ab81c3def78" (UID: "21239ee1-9c9e-44e2-ab73-4ab81c3def78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:01.623471 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.623431    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:01.623642 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.623449    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:01.623642 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.623559    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:01.623642 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.623596    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:01.624856 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.624829    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-envoy" pod "21239ee1-9c9e-44e2-ab73-4ab81c3def78" (UID: "21239ee1-9c9e-44e2-ab73-4ab81c3def78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:01.624990 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.624968    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-kube-api-access-m87l5" pod "21239ee1-9c9e-44e2-ab73-4ab81c3def78" (UID: "21239ee1-9c9e-44e2-ab73-4ab81c3def78"). InnerVolumeSpecName "kube-api-access-m87l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:01.624990 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.624981    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-token" pod "21239ee1-9c9e-44e2-ab73-4ab81c3def78" (UID: "21239ee1-9c9e-44e2-ab73-4ab81c3def78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:01.625067 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.624992    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-podinfo" pod "21239ee1-9c9e-44e2-ab73-4ab81c3def78" (UID: "21239ee1-9c9e-44e2-ab73-4ab81c3def78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:06:01.679098 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.679078    4420 generic.go:356] "Generic (PLEG): container finished" podID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerID="3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957" exitCode=0
Oct 30 20:06:01.679194 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.679128    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"
Oct 30 20:06:01.679194 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.679172    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" event={"ID":"21239ee1-9c9e-44e2-ab73-4ab81c3def78","Type":"ContainerDied","Data":"3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957"}
Oct 30 20:06:01.679306 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.679224    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d" event={"ID":"21239ee1-9c9e-44e2-ab73-4ab81c3def78","Type":"ContainerDied","Data":"120e292fa4593ee1caa2386706c8d626814f9202b6453b863d0ed128c07e53d9"}
Oct 30 20:06:01.679306 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.679235    4420 scope.go:122] "RemoveContainer" containerID="3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957"
Oct 30 20:06:01.680570 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.680546    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx" event={"ID":"bcd26faa-70f1-4b7f-9f3f-a632fdc0508c","Type":"ContainerDied","Data":"4567bdf74a04f2a7ed4dee8e502eee68609747e2ccd57fd5f014f45a16934021"}
Oct 30 20:06:01.680675 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.680592    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4567bdf74a04f2a7ed4dee8e502eee68609747e2ccd57fd5f014f45a16934021"
Oct 30 20:06:01.680675 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.680560    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"
Oct 30 20:06:01.688659 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.688640    4420 scope.go:122] "RemoveContainer" containerID="6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951"
Oct 30 20:06:01.696507 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.696488    4420 scope.go:122] "RemoveContainer" containerID="3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957"
Oct 30 20:06:01.696764 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:06:01.696745    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957\": container with ID starting with 3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957 not found: ID does not exist" containerID="3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957"
Oct 30 20:06:01.696845 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.696771    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957"} err="failed to get container status \"3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957\": rpc error: code = NotFound desc = could not find container \"3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957\": container with ID starting with 3523dc3273f312a5ee3623d384e42c6a8fa39698cac0eaddda4fedeaa6a38957 not found: ID does not exist"
Oct 30 20:06:01.696845 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.696785    4420 scope.go:122] "RemoveContainer" containerID="6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951"
Oct 30 20:06:01.697087 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:06:01.697069    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951\": container with ID starting with 6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951 not found: ID does not exist" containerID="6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951"
Oct 30 20:06:01.697125 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.697092    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951"} err="failed to get container status \"6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951\": rpc error: code = NotFound desc = could not find container \"6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951\": container with ID starting with 6dbdd51d2cac7eb9692ebc0bcb857a81e18b994f22f2259c0dd56a741e773951 not found: ID does not exist"
Oct 30 20:06:01.724361 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.724339    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m87l5\" (UniqueName: \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-kube-api-access-m87l5\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:01.724452 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.724365    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:01.724452 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.724377    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:01.724452 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.724390    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:01.724452 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.724405    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21239ee1-9c9e-44e2-ab73-4ab81c3def78-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:01.809722 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.809689    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"]
Oct 30 20:06:01.885223 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.885193    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"]
Oct 30 20:06:01.888740 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:01.888716    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f8a9d8-openshift-default-f4674b869-x6k5d"]
Oct 30 20:06:02.894690 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:02.894660    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" path="/var/lib/kubelet/pods/21239ee1-9c9e-44e2-ab73-4ab81c3def78/volumes"
Oct 30 20:06:05.195594 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:06:05.195516    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:06:12.175961 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.175929    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"]
Oct 30 20:06:12.176362 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.176348    4420 cpu_manager.go:414] "containerMap: removing container" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy"
Oct 30 20:06:12.176413 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.176365    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy"
Oct 30 20:06:12.176413 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.176400    4420 cpu_manager.go:414] "containerMap: removing container" podUID="bcd26faa-70f1-4b7f-9f3f-a632fdc0508c" containerName="revoke-keys"
Oct 30 20:06:12.176413 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.176405    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd26faa-70f1-4b7f-9f3f-a632fdc0508c" containerName="revoke-keys"
Oct 30 20:06:12.176517 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.176474    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy"
Oct 30 20:06:12.176517 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.176483    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd26faa-70f1-4b7f-9f3f-a632fdc0508c" containerName="revoke-keys"
Oct 30 20:06:12.176517 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.176491    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy"
Oct 30 20:06:12.181695 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.181679    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:12.215533 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.215506    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e67581bd-2vn5c\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:12.215643 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.215552    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9cmz\" (UniqueName: \"kubernetes.io/projected/8b4ee62b-dd50-4e0d-adfa-63403ae44527-kube-api-access-n9cmz\") pod \"maas-api-revoke-sub-e67581bd-2vn5c\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:12.215687 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.215634    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e67581bd-2vn5c\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:12.235678 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.235661    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:06:12.238369 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.238348    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"]
Oct 30 20:06:12.284970 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.284947    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:06:12.316359 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.316339    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e67581bd-2vn5c\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:12.316481 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.316384    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9cmz\" (UniqueName: \"kubernetes.io/projected/8b4ee62b-dd50-4e0d-adfa-63403ae44527-kube-api-access-n9cmz\") pod \"maas-api-revoke-sub-e67581bd-2vn5c\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:12.316481 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.316424    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e67581bd-2vn5c\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:12.317002 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.316984    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e67581bd-2vn5c\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:12.318999 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.318979    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e67581bd-2vn5c\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:12.372563 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.372536    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9cmz\" (UniqueName: \"kubernetes.io/projected/8b4ee62b-dd50-4e0d-adfa-63403ae44527-kube-api-access-n9cmz\") pod \"maas-api-revoke-sub-e67581bd-2vn5c\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:12.494354 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.494293    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:12.867157 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:12.867132    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"]
Oct 30 20:06:12.869476 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:06:12.869450    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4ee62b_dd50_4e0d_adfa_63403ae44527.slice/crio-cdb0e60f8d60faae5c6f887c2cfc9bc3873806696f14980f1ad9b925ee68ba97 WatchSource:0}: Error finding container cdb0e60f8d60faae5c6f887c2cfc9bc3873806696f14980f1ad9b925ee68ba97: Status 404 returned error can't find the container with id cdb0e60f8d60faae5c6f887c2cfc9bc3873806696f14980f1ad9b925ee68ba97
Oct 30 20:06:13.739555 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:13.739520    4420 generic.go:356] "Generic (PLEG): container finished" podID="8b4ee62b-dd50-4e0d-adfa-63403ae44527" containerID="684207f110fda394faffa327f781d0b81f8632c6197bb3be1e63804533d126f7" exitCode=0
Oct 30 20:06:13.739911 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:13.739575    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c" event={"ID":"8b4ee62b-dd50-4e0d-adfa-63403ae44527","Type":"ContainerDied","Data":"684207f110fda394faffa327f781d0b81f8632c6197bb3be1e63804533d126f7"}
Oct 30 20:06:13.739911 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:13.739609    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c" event={"ID":"8b4ee62b-dd50-4e0d-adfa-63403ae44527","Type":"ContainerStarted","Data":"cdb0e60f8d60faae5c6f887c2cfc9bc3873806696f14980f1ad9b925ee68ba97"}
Oct 30 20:06:14.881029 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:14.881006    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:14.941439 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:14.941414    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b4ee62b-dd50-4e0d-adfa-63403ae44527-kube-api-access-n9cmz\" (UniqueName: \"kubernetes.io/projected/8b4ee62b-dd50-4e0d-adfa-63403ae44527-kube-api-access-n9cmz\") pod \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") "
Oct 30 20:06:14.941567 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:14.941466    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-xks-ca\") pod \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") "
Oct 30 20:06:14.941567 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:14.941548    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-openshift-ca\") pod \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\" (UID: \"8b4ee62b-dd50-4e0d-adfa-63403ae44527\") "
Oct 30 20:06:14.941918 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:14.941896    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-openshift-ca" pod "8b4ee62b-dd50-4e0d-adfa-63403ae44527" (UID: "8b4ee62b-dd50-4e0d-adfa-63403ae44527"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:14.943502 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:14.943486    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-xks-ca" pod "8b4ee62b-dd50-4e0d-adfa-63403ae44527" (UID: "8b4ee62b-dd50-4e0d-adfa-63403ae44527"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:14.943601 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:14.943544    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4ee62b-dd50-4e0d-adfa-63403ae44527-kube-api-access-n9cmz" pod "8b4ee62b-dd50-4e0d-adfa-63403ae44527" (UID: "8b4ee62b-dd50-4e0d-adfa-63403ae44527"). InnerVolumeSpecName "kube-api-access-n9cmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:15.042783 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:15.042732    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9cmz\" (UniqueName: \"kubernetes.io/projected/8b4ee62b-dd50-4e0d-adfa-63403ae44527-kube-api-access-n9cmz\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:15.042783 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:15.042755    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:15.042783 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:15.042764    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b4ee62b-dd50-4e0d-adfa-63403ae44527-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:15.749798 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:15.749769    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"
Oct 30 20:06:15.749985 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:15.749774    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c" event={"ID":"8b4ee62b-dd50-4e0d-adfa-63403ae44527","Type":"ContainerDied","Data":"cdb0e60f8d60faae5c6f887c2cfc9bc3873806696f14980f1ad9b925ee68ba97"}
Oct 30 20:06:15.749985 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:15.749864    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdb0e60f8d60faae5c6f887c2cfc9bc3873806696f14980f1ad9b925ee68ba97"
Oct 30 20:06:16.829998 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:16.829946    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" containerID="cri-o://ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1" gracePeriod=45
Oct 30 20:06:17.078176 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.078147    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 20:06:17.161359 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.161329    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tls-certs\") pod \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") "
Oct 30 20:06:17.161508 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.161394    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-model-cache\") pod \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") "
Oct 30 20:06:17.161508 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.161430    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-dshm\") pod \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") "
Oct 30 20:06:17.161508 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.161457    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-kube-api-access-q7dxk\" (UniqueName: \"kubernetes.io/projected/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-kube-api-access-q7dxk\") pod \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") "
Oct 30 20:06:17.161702 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.161513    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tmp-dir\") pod \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") "
Oct 30 20:06:17.161702 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.161573    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-home\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-home\") pod \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\" (UID: \"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b\") "
Oct 30 20:06:17.161702 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.161648    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-model-cache" pod "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" (UID: "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:17.161905 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.161803    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tmp-dir" pod "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" (UID: "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:17.161905 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.161861    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-home" pod "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" (UID: "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:17.162012 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.161998    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.162050 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.162015    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.162050 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.162025    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.163749 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.163725    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-dshm" pod "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" (UID: "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:17.163749 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.163733    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tls-certs" pod "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" (UID: "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:17.163942 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.163863    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-kube-api-access-q7dxk" pod "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" (UID: "0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b"). InnerVolumeSpecName "kube-api-access-q7dxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:17.263066 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.263039    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.263066 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.263066    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.263256 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.263079    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7dxk\" (UniqueName: \"kubernetes.io/projected/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b-kube-api-access-q7dxk\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:17.767737 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.767698    4420 generic.go:356] "Generic (PLEG): container finished" podID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerID="ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1" exitCode=0
Oct 30 20:06:17.767918 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.767760    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"
Oct 30 20:06:17.767918 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.767782    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" event={"ID":"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b","Type":"ContainerDied","Data":"ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1"}
Oct 30 20:06:17.767918 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.767844    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" event={"ID":"0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b","Type":"ContainerDied","Data":"f9bb244e209f097c19152beac576ff5d053b1edbc4a48bd72cadee9bfb15eece"}
Oct 30 20:06:17.767918 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.767862    4420 scope.go:122] "RemoveContainer" containerID="ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1"
Oct 30 20:06:17.777108 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.777092    4420 scope.go:122] "RemoveContainer" containerID="ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1"
Oct 30 20:06:17.777354 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:06:17.777338    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1\": container with ID starting with ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1 not found: ID does not exist" containerID="ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1"
Oct 30 20:06:17.777401 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.777361    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1"} err="failed to get container status \"ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1\": rpc error: code = NotFound desc = could not find container \"ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1\": container with ID starting with ac0dd049c68e7f71ec24a7e78db4328cdb64156ea53693f392197fc46659c4c1 not found: ID does not exist"
Oct 30 20:06:17.929486 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.929461    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"]
Oct 30 20:06:17.978560 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:17.978539    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn"]
Oct 30 20:06:18.009340 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:18.009317    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b5178f/premium-w4-b5178f-kserve-899fd984c-zrkmn" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 30 20:06:18.894428 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:18.894401    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" path="/var/lib/kubelet/pods/0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b/volumes"
Oct 30 20:06:29.573642 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:29.573598    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"]
Oct 30 20:06:29.622611 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:29.622583    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b5178f-v5tjx"]
Oct 30 20:06:30.895074 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:30.895038    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd26faa-70f1-4b7f-9f3f-a632fdc0508c" path="/var/lib/kubelet/pods/bcd26faa-70f1-4b7f-9f3f-a632fdc0508c/volumes"
Oct 30 20:06:35.196495 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:06:35.196452    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:06:35.340802 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.340775    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"]
Oct 30 20:06:35.341202 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.341189    4420 cpu_manager.go:414] "containerMap: removing container" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main"
Oct 30 20:06:35.341250 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.341205    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main"
Oct 30 20:06:35.341250 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.341218    4420 cpu_manager.go:414] "containerMap: removing container" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy"
Oct 30 20:06:35.341250 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.341223    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21239ee1-9c9e-44e2-ab73-4ab81c3def78" containerName="istio-proxy"
Oct 30 20:06:35.341250 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.341234    4420 cpu_manager.go:414] "containerMap: removing container" podUID="8b4ee62b-dd50-4e0d-adfa-63403ae44527" containerName="revoke-keys"
Oct 30 20:06:35.341250 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.341239    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b4ee62b-dd50-4e0d-adfa-63403ae44527" containerName="revoke-keys"
Oct 30 20:06:35.341394 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.341330    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0e5afb-30a8-41e6-bc35-9c5e2f9b488b" containerName="main"
Oct 30 20:06:35.341394 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.341339    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b4ee62b-dd50-4e0d-adfa-63403ae44527" containerName="revoke-keys"
Oct 30 20:06:35.344568 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.344551    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.396469 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.396449    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b5178f-openshift-default-dockercfg-k2spr\""
Oct 30 20:06:35.418925 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.418905    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.419028 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.418934    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.419028 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.418965    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.419102 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.419032    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5rpm\" (UniqueName: \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-kube-api-access-l5rpm\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.419196 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.419181    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.419231 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.419221    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.419266 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.419241    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.419303 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.419267    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.419347 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.419332    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.448236 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.448180    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"]
Oct 30 20:06:35.520221 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520188    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.520357 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520239    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.520357 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520266    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.520357 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520299    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.520357 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520326    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.520579 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520398    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.520579 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520426    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.520579 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520479    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.520745 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520600    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5rpm\" (UniqueName: \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-kube-api-access-l5rpm\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.520745 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520641    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-certs\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.520745 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520682    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.520745 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520731    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-data\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.521008 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.520942    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-credential-socket\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.521222 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.521155    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istiod-ca-cert\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.522856 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.522839    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-podinfo\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.522957 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.522937    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-envoy\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.576721 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.576701    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-token\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.576828 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.576798    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5rpm\" (UniqueName: \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-kube-api-access-l5rpm\") pod \"e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") " pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:35.657731 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:35.657713    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:36.078802 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:36.078730    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"]
Oct 30 20:06:36.082338 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:06:36.082313    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7e6ed0_eacb_44b8_bebe_4332e5856f2a.slice/crio-40eec12a208e88eb48409274bcdbd1964619ea3d104ebcbb2b83da395605848e WatchSource:0}: Error finding container 40eec12a208e88eb48409274bcdbd1964619ea3d104ebcbb2b83da395605848e: Status 404 returned error can't find the container with id 40eec12a208e88eb48409274bcdbd1964619ea3d104ebcbb2b83da395605848e
Oct 30 20:06:36.084286 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:36.084253    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:06:36.084374 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:36.084316    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:06:36.084374 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:36.084346    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:06:36.858574 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:36.858535    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd" event={"ID":"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a","Type":"ContainerStarted","Data":"7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92"}
Oct 30 20:06:36.858978 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:36.858610    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd" event={"ID":"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a","Type":"ContainerStarted","Data":"40eec12a208e88eb48409274bcdbd1964619ea3d104ebcbb2b83da395605848e"}
Oct 30 20:06:36.962197 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:36.962165    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"]
Oct 30 20:06:37.007030 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:37.006984    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd" podStartSLOduration=2.006969491 podStartE2EDuration="2.006969491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:37.005076709 +0000 UTC m=+3044.216293850" watchObservedRunningTime="2026-10-30 20:06:37.006969491 +0000 UTC m=+3044.218186635"
Oct 30 20:06:37.657873 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:37.657846    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:37.662910 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:37.662889    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:37.863455 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:37.863425    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:37.863911 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:37.863434    4420 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd" secret="" err="secret \"e2e-worker-w4-b5178f-openshift-default-dockercfg-k2spr\" not found"
Oct 30 20:06:37.864352 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:37.864335    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:38.869248 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:38.869217    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd" podUID="4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" containerName="istio-proxy" containerID="cri-o://7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92" gracePeriod=30
Oct 30 20:06:42.178527 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:42.178499    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-fkfr5"]
Oct 30 20:06:42.887933 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:42.887883    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5" podUID="6981f9c5-0fa0-4050-8b66-62b07f42ab85" containerName="manager" containerID="cri-o://936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27" gracePeriod=30
Oct 30 20:06:43.335689 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.335667    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 20:06:43.492630 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.492559    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-metrics-certs\") pod \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") "
Oct 30 20:06:43.492630 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.492607    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6981f9c5-0fa0-4050-8b66-62b07f42ab85-kube-api-access-s8l45\" (UniqueName: \"kubernetes.io/projected/6981f9c5-0fa0-4050-8b66-62b07f42ab85-kube-api-access-s8l45\") pod \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") "
Oct 30 20:06:43.492837 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.492664    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-webhook-cert\") pod \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\" (UID: \"6981f9c5-0fa0-4050-8b66-62b07f42ab85\") "
Oct 30 20:06:43.494649 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.494620    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-metrics-certs" pod "6981f9c5-0fa0-4050-8b66-62b07f42ab85" (UID: "6981f9c5-0fa0-4050-8b66-62b07f42ab85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:43.494875 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.494842    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-webhook-cert" pod "6981f9c5-0fa0-4050-8b66-62b07f42ab85" (UID: "6981f9c5-0fa0-4050-8b66-62b07f42ab85"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:06:43.494985 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.494925    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6981f9c5-0fa0-4050-8b66-62b07f42ab85-kube-api-access-s8l45" pod "6981f9c5-0fa0-4050-8b66-62b07f42ab85" (UID: "6981f9c5-0fa0-4050-8b66-62b07f42ab85"). InnerVolumeSpecName "kube-api-access-s8l45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:43.594078 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.594050    4420 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-webhook-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:43.594078 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.594074    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6981f9c5-0fa0-4050-8b66-62b07f42ab85-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:43.594232 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.594084    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8l45\" (UniqueName: \"kubernetes.io/projected/6981f9c5-0fa0-4050-8b66-62b07f42ab85-kube-api-access-s8l45\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:43.895112 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.895086    4420 generic.go:356] "Generic (PLEG): container finished" podID="6981f9c5-0fa0-4050-8b66-62b07f42ab85" containerID="936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27" exitCode=0
Oct 30 20:06:43.895215 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.895131    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5" event={"ID":"6981f9c5-0fa0-4050-8b66-62b07f42ab85","Type":"ContainerDied","Data":"936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27"}
Oct 30 20:06:43.895215 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.895144    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5"
Oct 30 20:06:43.895215 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.895167    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5d5b686b-fkfr5" event={"ID":"6981f9c5-0fa0-4050-8b66-62b07f42ab85","Type":"ContainerDied","Data":"500d137378eb453b37d0a295584cb9f1ae4ebc7cce263d027cbe4f934f56bd35"}
Oct 30 20:06:43.895215 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.895187    4420 scope.go:122] "RemoveContainer" containerID="936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27"
Oct 30 20:06:43.979607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.979589    4420 scope.go:122] "RemoveContainer" containerID="936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27"
Oct 30 20:06:43.979884 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:06:43.979864    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27\": container with ID starting with 936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27 not found: ID does not exist" containerID="936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27"
Oct 30 20:06:43.979942 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:43.979890    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27"} err="failed to get container status \"936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27\": rpc error: code = NotFound desc = could not find container \"936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27\": container with ID starting with 936fd0215003dbf32eb8f4a90d580f8daba5f85e82828e947fa6967b0fdf1d27 not found: ID does not exist"
Oct 30 20:06:44.058020 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.057988    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-fkfr5"]
Oct 30 20:06:44.060735 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.060708    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-fkfr5"]
Oct 30 20:06:44.106048 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.106026    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:44.198623 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.198594    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-envoy\") pod \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") "
Oct 30 20:06:44.198744 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.198640    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istiod-ca-cert\") pod \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") "
Oct 30 20:06:44.198744 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.198728    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-socket\") pod \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") "
Oct 30 20:06:44.198904 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.198850    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-kube-api-access-l5rpm\" (UniqueName: \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-kube-api-access-l5rpm\") pod \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") "
Oct 30 20:06:44.198904 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.198880    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-podinfo\") pod \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") "
Oct 30 20:06:44.199016 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.198912    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-data\") pod \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") "
Oct 30 20:06:44.199016 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.198939    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-credential-socket\") pod \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") "
Oct 30 20:06:44.199016 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.198974    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-certs\") pod \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") "
Oct 30 20:06:44.199177 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.199019    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-socket" pod "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" (UID: "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:44.199177 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.199049    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istiod-ca-cert" pod "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" (UID: "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:06:44.199177 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.199090    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-token\" (UniqueName: \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-token\") pod \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\" (UID: \"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a\") "
Oct 30 20:06:44.199324 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.199244    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-credential-socket" pod "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" (UID: "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:44.199427 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.199401    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-data" pod "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" (UID: "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:44.199539 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.199472    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-certs" pod "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" (UID: "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:44.199539 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.199527    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:44.199662 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.199547    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:44.199662 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.199561    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:44.199662 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.199575    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:44.201222 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.201196    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-envoy" pod "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" (UID: "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:06:44.201549 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.201531    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-podinfo" pod "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" (UID: "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:06:44.201849 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.201802    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-kube-api-access-l5rpm" pod "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" (UID: "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a"). InnerVolumeSpecName "kube-api-access-l5rpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:44.201958 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.201920    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-token" pod "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" (UID: "4e7e6ed0-eacb-44b8-bebe-4332e5856f2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:06:44.300800 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.300763    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5rpm\" (UniqueName: \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-kube-api-access-l5rpm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:44.300800 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.300787    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:44.300800 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.300797    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:44.301114 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.300826    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:44.301114 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.300839    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:06:44.897421 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.897383    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6981f9c5-0fa0-4050-8b66-62b07f42ab85" path="/var/lib/kubelet/pods/6981f9c5-0fa0-4050-8b66-62b07f42ab85/volumes"
Oct 30 20:06:44.902324 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.902296    4420 generic.go:356] "Generic (PLEG): container finished" podID="4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" containerID="7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92" exitCode=0
Oct 30 20:06:44.902472 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.902352    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"
Oct 30 20:06:44.902472 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.902393    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd" event={"ID":"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a","Type":"ContainerDied","Data":"7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92"}
Oct 30 20:06:44.902472 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.902459    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd" event={"ID":"4e7e6ed0-eacb-44b8-bebe-4332e5856f2a","Type":"ContainerDied","Data":"40eec12a208e88eb48409274bcdbd1964619ea3d104ebcbb2b83da395605848e"}
Oct 30 20:06:44.902644 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.902498    4420 scope.go:122] "RemoveContainer" containerID="7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92"
Oct 30 20:06:44.921953 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.921931    4420 scope.go:122] "RemoveContainer" containerID="7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92"
Oct 30 20:06:44.922255 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:06:44.922234    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92\": container with ID starting with 7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92 not found: ID does not exist" containerID="7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92"
Oct 30 20:06:44.922337 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:44.922260    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92"} err="failed to get container status \"7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92\": rpc error: code = NotFound desc = could not find container \"7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92\": container with ID starting with 7cb699ba60b2ec06c71bb520ce32e26b705e95fbeee4ae3c71d7e38553c43e92 not found: ID does not exist"
Oct 30 20:06:45.066634 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:45.066605    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"]
Oct 30 20:06:45.116229 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:45.116196    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b5178f-openshift-default-c8fbdff8c-sd4vd"]
Oct 30 20:06:46.895376 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:46.895340    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" path="/var/lib/kubelet/pods/4e7e6ed0-eacb-44b8-bebe-4332e5856f2a/volumes"
Oct 30 20:06:53.161048 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.161016    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-zm9pq"]
Oct 30 20:06:53.161476 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.161416    4420 cpu_manager.go:414] "containerMap: removing container" podUID="6981f9c5-0fa0-4050-8b66-62b07f42ab85" containerName="manager"
Oct 30 20:06:53.161476 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.161427    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6981f9c5-0fa0-4050-8b66-62b07f42ab85" containerName="manager"
Oct 30 20:06:53.161476 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.161448    4420 cpu_manager.go:414] "containerMap: removing container" podUID="4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" containerName="istio-proxy"
Oct 30 20:06:53.161476 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.161453    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" containerName="istio-proxy"
Oct 30 20:06:53.161611 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.161511    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7e6ed0-eacb-44b8-bebe-4332e5856f2a" containerName="istio-proxy"
Oct 30 20:06:53.161611 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.161525    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="6981f9c5-0fa0-4050-8b66-62b07f42ab85" containerName="manager"
Oct 30 20:06:53.168507 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.168473    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:53.198345 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.198320    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqrtb\" (UniqueName: \"kubernetes.io/projected/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-kube-api-access-cqrtb\") pod \"maas-controller-6b5d5b686b-zm9pq\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") " pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:53.198506 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.198385    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-webhook-cert\") pod \"maas-controller-6b5d5b686b-zm9pq\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") " pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:53.198506 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.198479    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-metrics-certs\") pod \"maas-controller-6b5d5b686b-zm9pq\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") " pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:53.220663 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.220641    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 20:06:53.269905 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.269886    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 20:06:53.270018 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.269962    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 20:06:53.270018 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.269959    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sq2hw\""
Oct 30 20:06:53.271942 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.271921    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-zm9pq"]
Oct 30 20:06:53.299507 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.299480    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-webhook-cert\") pod \"maas-controller-6b5d5b686b-zm9pq\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") " pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:53.299623 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.299526    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-metrics-certs\") pod \"maas-controller-6b5d5b686b-zm9pq\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") " pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:53.299623 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.299575    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqrtb\" (UniqueName: \"kubernetes.io/projected/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-kube-api-access-cqrtb\") pod \"maas-controller-6b5d5b686b-zm9pq\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") " pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:53.301956 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.301933    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-webhook-cert\") pod \"maas-controller-6b5d5b686b-zm9pq\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") " pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:53.302075 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.301991    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-metrics-certs\") pod \"maas-controller-6b5d5b686b-zm9pq\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") " pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:53.355946 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.355921    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqrtb\" (UniqueName: \"kubernetes.io/projected/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-kube-api-access-cqrtb\") pod \"maas-controller-6b5d5b686b-zm9pq\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") " pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:53.482109 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.482064    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:53.761743 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.761680    4420 scope.go:122] "RemoveContainer" containerID="056955c3e89779f0323ca8583770599eb944211b6a82eeddeebc39136007943f"
Oct 30 20:06:53.848974 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.848950    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-zm9pq"]
Oct 30 20:06:53.881916 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:06:53.881893    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bd6c17_51b4_46c2_be02_a2b7a2c0b8f0.slice/crio-46aabf0720b018b908c21cdc7f7c8bac15763dfcb6cfa6daf61d246af7e91ec8 WatchSource:0}: Error finding container 46aabf0720b018b908c21cdc7f7c8bac15763dfcb6cfa6daf61d246af7e91ec8: Status 404 returned error can't find the container with id 46aabf0720b018b908c21cdc7f7c8bac15763dfcb6cfa6daf61d246af7e91ec8
Oct 30 20:06:53.883109 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.883093    4420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:06:53.964171 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:53.964147    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq" event={"ID":"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0","Type":"ContainerStarted","Data":"46aabf0720b018b908c21cdc7f7c8bac15763dfcb6cfa6daf61d246af7e91ec8"}
Oct 30 20:06:54.970170 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:54.970134    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq" event={"ID":"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0","Type":"ContainerStarted","Data":"2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab"}
Oct 30 20:06:54.970535 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:54.970258    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:06:55.085679 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:06:55.085604    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq" podStartSLOduration=1.205797841 podStartE2EDuration="2.085590003s" totalImagesPullingTime="879.792162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:55.083701465 +0000 UTC m=+3062.294918609" watchObservedRunningTime="2026-10-30 20:06:55.085590003 +0000 UTC m=+3062.296807171"
Oct 30 20:07:05.196112 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:07:05.196076    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:07:05.977059 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:05.977030    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:07:15.842550 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:15.842519    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"]
Oct 30 20:07:15.846304 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:15.846282    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:15.898141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:15.898118    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:07:15.947381 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:15.947350    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:07:15.949241 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:15.949214    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"]
Oct 30 20:07:15.992353 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:15.992321    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fqjwt\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:15.992523 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:15.992357    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fqjwt\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:15.992523 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:15.992473    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nljj2\" (UniqueName: \"kubernetes.io/projected/c788fa3f-ff2c-4756-9fc0-de975d013aa3-kube-api-access-nljj2\") pod \"maas-api-revoke-sub-85237e45-fqjwt\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:16.093732 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:16.093659    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fqjwt\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:16.093732 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:16.093694    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fqjwt\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:16.093941 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:16.093756    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nljj2\" (UniqueName: \"kubernetes.io/projected/c788fa3f-ff2c-4756-9fc0-de975d013aa3-kube-api-access-nljj2\") pod \"maas-api-revoke-sub-85237e45-fqjwt\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:16.094582 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:16.094468    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-fqjwt\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:16.096230 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:16.096209    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-fqjwt\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:16.150788 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:16.150766    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nljj2\" (UniqueName: \"kubernetes.io/projected/c788fa3f-ff2c-4756-9fc0-de975d013aa3-kube-api-access-nljj2\") pod \"maas-api-revoke-sub-85237e45-fqjwt\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:16.159765 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:16.159743    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:16.534875 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:16.534851    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"]
Oct 30 20:07:16.536684 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:07:16.536659    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc788fa3f_ff2c_4756_9fc0_de975d013aa3.slice/crio-9f968639f207bd59c935bc74a70d5ecdb6dfac912f181f603a1458f317226ed2 WatchSource:0}: Error finding container 9f968639f207bd59c935bc74a70d5ecdb6dfac912f181f603a1458f317226ed2: Status 404 returned error can't find the container with id 9f968639f207bd59c935bc74a70d5ecdb6dfac912f181f603a1458f317226ed2
Oct 30 20:07:17.068348 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:17.068274    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt" event={"ID":"c788fa3f-ff2c-4756-9fc0-de975d013aa3","Type":"ContainerStarted","Data":"a7dd354eff61a2623056a3594c4d0da2afd82fc3db4b75408cb7d8d2d4e23b23"}
Oct 30 20:07:17.068348 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:17.068332    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt" event={"ID":"c788fa3f-ff2c-4756-9fc0-de975d013aa3","Type":"ContainerStarted","Data":"9f968639f207bd59c935bc74a70d5ecdb6dfac912f181f603a1458f317226ed2"}
Oct 30 20:07:17.183862 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:17.183796    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt" podStartSLOduration=2.183782177 podStartE2EDuration="2.183782177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:17.183565239 +0000 UTC m=+3084.394782383" watchObservedRunningTime="2026-10-30 20:07:17.183782177 +0000 UTC m=+3084.394999322"
Oct 30 20:07:18.074135 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:18.074096    4420 generic.go:356] "Generic (PLEG): container finished" podID="c788fa3f-ff2c-4756-9fc0-de975d013aa3" containerID="a7dd354eff61a2623056a3594c4d0da2afd82fc3db4b75408cb7d8d2d4e23b23" exitCode=0
Oct 30 20:07:18.074490 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:18.074154    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt" event={"ID":"c788fa3f-ff2c-4756-9fc0-de975d013aa3","Type":"ContainerDied","Data":"a7dd354eff61a2623056a3594c4d0da2afd82fc3db4b75408cb7d8d2d4e23b23"}
Oct 30 20:07:19.214438 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:19.214409    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:19.324330 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:19.324301    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c788fa3f-ff2c-4756-9fc0-de975d013aa3-kube-api-access-nljj2\" (UniqueName: \"kubernetes.io/projected/c788fa3f-ff2c-4756-9fc0-de975d013aa3-kube-api-access-nljj2\") pod \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") "
Oct 30 20:07:19.324492 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:19.324471    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-openshift-ca\") pod \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") "
Oct 30 20:07:19.324561 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:19.324512    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-xks-ca\") pod \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\" (UID: \"c788fa3f-ff2c-4756-9fc0-de975d013aa3\") "
Oct 30 20:07:19.324838 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:19.324789    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-openshift-ca" pod "c788fa3f-ff2c-4756-9fc0-de975d013aa3" (UID: "c788fa3f-ff2c-4756-9fc0-de975d013aa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:07:19.326656 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:19.326616    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c788fa3f-ff2c-4756-9fc0-de975d013aa3-kube-api-access-nljj2" pod "c788fa3f-ff2c-4756-9fc0-de975d013aa3" (UID: "c788fa3f-ff2c-4756-9fc0-de975d013aa3"). InnerVolumeSpecName "kube-api-access-nljj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:19.326975 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:19.326807    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-xks-ca" pod "c788fa3f-ff2c-4756-9fc0-de975d013aa3" (UID: "c788fa3f-ff2c-4756-9fc0-de975d013aa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:19.425373 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:19.425341    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:19.425491 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:19.425376    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c788fa3f-ff2c-4756-9fc0-de975d013aa3-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:19.425491 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:19.425393    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nljj2\" (UniqueName: \"kubernetes.io/projected/c788fa3f-ff2c-4756-9fc0-de975d013aa3-kube-api-access-nljj2\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:20.087410 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:20.087357    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt" event={"ID":"c788fa3f-ff2c-4756-9fc0-de975d013aa3","Type":"ContainerDied","Data":"9f968639f207bd59c935bc74a70d5ecdb6dfac912f181f603a1458f317226ed2"}
Oct 30 20:07:20.087627 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:20.087435    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f968639f207bd59c935bc74a70d5ecdb6dfac912f181f603a1458f317226ed2"
Oct 30 20:07:20.087627 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:20.087499    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"
Oct 30 20:07:21.040947 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:21.040911    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-748lj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:07:21.041413 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:21.040972    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:07:27.777768 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:27.777732    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-zm9pq"]
Oct 30 20:07:27.778273 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:27.777933    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq" podUID="93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0" containerName="manager" containerID="cri-o://2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab" gracePeriod=30
Oct 30 20:07:28.022108 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.022081    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:07:28.097910 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.097854    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-kube-api-access-cqrtb\" (UniqueName: \"kubernetes.io/projected/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-kube-api-access-cqrtb\") pod \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") "
Oct 30 20:07:28.098041 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.097918    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-metrics-certs\") pod \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") "
Oct 30 20:07:28.098041 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.097965    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-webhook-cert\") pod \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\" (UID: \"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0\") "
Oct 30 20:07:28.099758 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.099734    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-metrics-certs" pod "93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0" (UID: "93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:28.100035 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.100010    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-webhook-cert" pod "93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0" (UID: "93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:07:28.100145 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.100042    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-kube-api-access-cqrtb" pod "93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0" (UID: "93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0"). InnerVolumeSpecName "kube-api-access-cqrtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:07:28.126882 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.126855    4420 generic.go:356] "Generic (PLEG): container finished" podID="93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0" containerID="2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab" exitCode=0
Oct 30 20:07:28.126974 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.126901    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq"
Oct 30 20:07:28.126974 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.126943    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq" event={"ID":"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0","Type":"ContainerDied","Data":"2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab"}
Oct 30 20:07:28.127048 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.126997    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5d5b686b-zm9pq" event={"ID":"93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0","Type":"ContainerDied","Data":"46aabf0720b018b908c21cdc7f7c8bac15763dfcb6cfa6daf61d246af7e91ec8"}
Oct 30 20:07:28.127048 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.127024    4420 scope.go:122] "RemoveContainer" containerID="2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab"
Oct 30 20:07:28.140280 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.140260    4420 scope.go:122] "RemoveContainer" containerID="2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab"
Oct 30 20:07:28.140603 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:07:28.140576    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab\": container with ID starting with 2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab not found: ID does not exist" containerID="2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab"
Oct 30 20:07:28.140695 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.140609    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab"} err="failed to get container status \"2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab\": rpc error: code = NotFound desc = could not find container \"2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab\": container with ID starting with 2e403c9be9f7dced34263813eb6c278d9f19a88af9aa5fdc16d504c0890c64ab not found: ID does not exist"
Oct 30 20:07:28.199129 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.199099    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqrtb\" (UniqueName: \"kubernetes.io/projected/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-kube-api-access-cqrtb\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:28.199129 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.199123    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:28.199129 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.199133    4420 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0-webhook-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:07:28.244161 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.244087    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-zm9pq"]
Oct 30 20:07:28.247106 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.247075    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-zm9pq"]
Oct 30 20:07:28.893979 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:28.893949    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0" path="/var/lib/kubelet/pods/93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0/volumes"
Oct 30 20:07:29.625781 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.625747    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-r9n2j"]
Oct 30 20:07:29.626172 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.626159    4420 cpu_manager.go:414] "containerMap: removing container" podUID="c788fa3f-ff2c-4756-9fc0-de975d013aa3" containerName="revoke-keys"
Oct 30 20:07:29.626226 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.626174    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c788fa3f-ff2c-4756-9fc0-de975d013aa3" containerName="revoke-keys"
Oct 30 20:07:29.626226 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.626185    4420 cpu_manager.go:414] "containerMap: removing container" podUID="93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0" containerName="manager"
Oct 30 20:07:29.626226 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.626189    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0" containerName="manager"
Oct 30 20:07:29.626329 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.626253    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bd6c17-51b4-46c2-be02-a2b7a2c0b8f0" containerName="manager"
Oct 30 20:07:29.626329 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.626269    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c788fa3f-ff2c-4756-9fc0-de975d013aa3" containerName="revoke-keys"
Oct 30 20:07:29.630452 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.630434    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:29.682524 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.682500    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 20:07:29.711328 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.711305    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24780d87-6614-4bf9-a587-89e043a10f55-metrics-certs\") pod \"maas-controller-6b5d5b686b-r9n2j\" (UID: \"24780d87-6614-4bf9-a587-89e043a10f55\") " pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:29.711432 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.711353    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p268c\" (UniqueName: \"kubernetes.io/projected/24780d87-6614-4bf9-a587-89e043a10f55-kube-api-access-p268c\") pod \"maas-controller-6b5d5b686b-r9n2j\" (UID: \"24780d87-6614-4bf9-a587-89e043a10f55\") " pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:29.711478 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.711439    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24780d87-6614-4bf9-a587-89e043a10f55-webhook-cert\") pod \"maas-controller-6b5d5b686b-r9n2j\" (UID: \"24780d87-6614-4bf9-a587-89e043a10f55\") " pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:29.731591 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.731555    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 20:07:29.731698 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.731590    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sq2hw\""
Oct 30 20:07:29.731698 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.731613    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 20:07:29.734141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.734117    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-r9n2j"]
Oct 30 20:07:29.812090 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.812043    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24780d87-6614-4bf9-a587-89e043a10f55-metrics-certs\") pod \"maas-controller-6b5d5b686b-r9n2j\" (UID: \"24780d87-6614-4bf9-a587-89e043a10f55\") " pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:29.812261 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.812096    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p268c\" (UniqueName: \"kubernetes.io/projected/24780d87-6614-4bf9-a587-89e043a10f55-kube-api-access-p268c\") pod \"maas-controller-6b5d5b686b-r9n2j\" (UID: \"24780d87-6614-4bf9-a587-89e043a10f55\") " pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:29.812261 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.812129    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24780d87-6614-4bf9-a587-89e043a10f55-webhook-cert\") pod \"maas-controller-6b5d5b686b-r9n2j\" (UID: \"24780d87-6614-4bf9-a587-89e043a10f55\") " pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:29.814669 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.814640    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/24780d87-6614-4bf9-a587-89e043a10f55-webhook-cert\") pod \"maas-controller-6b5d5b686b-r9n2j\" (UID: \"24780d87-6614-4bf9-a587-89e043a10f55\") " pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:29.814890 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.814868    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24780d87-6614-4bf9-a587-89e043a10f55-metrics-certs\") pod \"maas-controller-6b5d5b686b-r9n2j\" (UID: \"24780d87-6614-4bf9-a587-89e043a10f55\") " pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:29.869211 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.869185    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p268c\" (UniqueName: \"kubernetes.io/projected/24780d87-6614-4bf9-a587-89e043a10f55-kube-api-access-p268c\") pod \"maas-controller-6b5d5b686b-r9n2j\" (UID: \"24780d87-6614-4bf9-a587-89e043a10f55\") " pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:29.942733 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:29.942713    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:30.310719 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:30.310693    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6b5d5b686b-r9n2j"]
Oct 30 20:07:30.312717 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:07:30.312687    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24780d87_6614_4bf9_a587_89e043a10f55.slice/crio-e6ac573c7f5963ba2534d9852ddae6e6930f0196b70b6f8aa679a7152d824811 WatchSource:0}: Error finding container e6ac573c7f5963ba2534d9852ddae6e6930f0196b70b6f8aa679a7152d824811: Status 404 returned error can't find the container with id e6ac573c7f5963ba2534d9852ddae6e6930f0196b70b6f8aa679a7152d824811
Oct 30 20:07:31.141998 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:31.141954    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5d5b686b-r9n2j" event={"ID":"24780d87-6614-4bf9-a587-89e043a10f55","Type":"ContainerStarted","Data":"e6ac573c7f5963ba2534d9852ddae6e6930f0196b70b6f8aa679a7152d824811"}
Oct 30 20:07:32.148787 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:32.148754    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6b5d5b686b-r9n2j" event={"ID":"24780d87-6614-4bf9-a587-89e043a10f55","Type":"ContainerStarted","Data":"4814a552dcdaca624e63efcfa32a97d823ba0599d7d5d62f8ca948d008a0ed98"}
Oct 30 20:07:32.149177 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:32.148929    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:07:32.263591 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:32.263530    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6b5d5b686b-r9n2j" podStartSLOduration=2.395059469 podStartE2EDuration="3.263512438s" totalImagesPullingTime="868.452969ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:32.262158143 +0000 UTC m=+3099.473375272" watchObservedRunningTime="2026-10-30 20:07:32.263512438 +0000 UTC m=+3099.474729584"
Oct 30 20:07:35.196573 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:07:35.196499    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:07:43.156018 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:07:43.155984    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6b5d5b686b-r9n2j"
Oct 30 20:08:05.195461 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:08:05.195426    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:08:35.196653 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:08:35.196617    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:08:47.539265 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.539237    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"]
Oct 30 20:08:47.543014 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.542997    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:47.584595 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.584572    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rgr4\" (UniqueName: \"kubernetes.io/projected/a350d00b-cf17-417f-b0d8-dd29dd979f51-kube-api-access-4rgr4\") pod \"maas-api-revoke-sub-c1e271a9-87k75\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:47.584704 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.584601    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-87k75\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:47.584751 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.584709    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-87k75\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:47.595171 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.595151    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:08:47.595437 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.595420    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:08:47.607120 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.602552    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"]
Oct 30 20:08:47.686302 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.686271    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-87k75\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:47.686468 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.686322    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rgr4\" (UniqueName: \"kubernetes.io/projected/a350d00b-cf17-417f-b0d8-dd29dd979f51-kube-api-access-4rgr4\") pod \"maas-api-revoke-sub-c1e271a9-87k75\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:47.686468 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.686449    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-87k75\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:47.686942 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.686909    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-87k75\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:47.688852 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.688831    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-87k75\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:47.744612 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.744585    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rgr4\" (UniqueName: \"kubernetes.io/projected/a350d00b-cf17-417f-b0d8-dd29dd979f51-kube-api-access-4rgr4\") pod \"maas-api-revoke-sub-c1e271a9-87k75\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:47.856263 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:47.856213    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:48.225161 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:48.225138    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"]
Oct 30 20:08:48.227344 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:08:48.227318    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda350d00b_cf17_417f_b0d8_dd29dd979f51.slice/crio-df23524c55f5915ac02a0e9c8772f25500820b5c6de0db50b8672d2a5ff63722 WatchSource:0}: Error finding container df23524c55f5915ac02a0e9c8772f25500820b5c6de0db50b8672d2a5ff63722: Status 404 returned error can't find the container with id df23524c55f5915ac02a0e9c8772f25500820b5c6de0db50b8672d2a5ff63722
Oct 30 20:08:48.515472 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:48.515417    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75" event={"ID":"a350d00b-cf17-417f-b0d8-dd29dd979f51","Type":"ContainerStarted","Data":"df23524c55f5915ac02a0e9c8772f25500820b5c6de0db50b8672d2a5ff63722"}
Oct 30 20:08:49.521084 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:49.521059    4420 generic.go:356] "Generic (PLEG): container finished" podID="a350d00b-cf17-417f-b0d8-dd29dd979f51" containerID="a1f5c2c359f0551f8e4ea2ae1021d12c0b18e15267ce9eaab2ad3d7bea485aeb" exitCode=0
Oct 30 20:08:49.521417 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:49.521143    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75" event={"ID":"a350d00b-cf17-417f-b0d8-dd29dd979f51","Type":"ContainerDied","Data":"a1f5c2c359f0551f8e4ea2ae1021d12c0b18e15267ce9eaab2ad3d7bea485aeb"}
Oct 30 20:08:50.666446 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:50.666425    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:08:50.711770 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:50.711747    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a350d00b-cf17-417f-b0d8-dd29dd979f51-kube-api-access-4rgr4\" (UniqueName: \"kubernetes.io/projected/a350d00b-cf17-417f-b0d8-dd29dd979f51-kube-api-access-4rgr4\") pod \"a350d00b-cf17-417f-b0d8-dd29dd979f51\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") "
Oct 30 20:08:50.711892 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:50.711850    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-xks-ca\") pod \"a350d00b-cf17-417f-b0d8-dd29dd979f51\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") "
Oct 30 20:08:50.711945 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:50.711935    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-openshift-ca\") pod \"a350d00b-cf17-417f-b0d8-dd29dd979f51\" (UID: \"a350d00b-cf17-417f-b0d8-dd29dd979f51\") "
Oct 30 20:08:50.712224 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:50.712205    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-openshift-ca" pod "a350d00b-cf17-417f-b0d8-dd29dd979f51" (UID: "a350d00b-cf17-417f-b0d8-dd29dd979f51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:08:50.712326 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:50.712312    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:50.713869 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:50.713847    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a350d00b-cf17-417f-b0d8-dd29dd979f51-kube-api-access-4rgr4" pod "a350d00b-cf17-417f-b0d8-dd29dd979f51" (UID: "a350d00b-cf17-417f-b0d8-dd29dd979f51"). InnerVolumeSpecName "kube-api-access-4rgr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:08:50.713959 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:50.713924    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-xks-ca" pod "a350d00b-cf17-417f-b0d8-dd29dd979f51" (UID: "a350d00b-cf17-417f-b0d8-dd29dd979f51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:08:50.813262 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:50.813202    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rgr4\" (UniqueName: \"kubernetes.io/projected/a350d00b-cf17-417f-b0d8-dd29dd979f51-kube-api-access-4rgr4\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:50.813262 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:50.813236    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a350d00b-cf17-417f-b0d8-dd29dd979f51-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:08:51.531403 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:51.531375    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75" event={"ID":"a350d00b-cf17-417f-b0d8-dd29dd979f51","Type":"ContainerDied","Data":"df23524c55f5915ac02a0e9c8772f25500820b5c6de0db50b8672d2a5ff63722"}
Oct 30 20:08:51.531567 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:51.531426    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df23524c55f5915ac02a0e9c8772f25500820b5c6de0db50b8672d2a5ff63722"
Oct 30 20:08:51.531567 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:08:51.531385    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"
Oct 30 20:09:05.195022 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:09:05.194943    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:09:32.932074 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:32.932044    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"]
Oct 30 20:09:32.932557 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:32.932433    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a350d00b-cf17-417f-b0d8-dd29dd979f51" containerName="revoke-keys"
Oct 30 20:09:32.932557 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:32.932443    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a350d00b-cf17-417f-b0d8-dd29dd979f51" containerName="revoke-keys"
Oct 30 20:09:32.932557 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:32.932505    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a350d00b-cf17-417f-b0d8-dd29dd979f51" containerName="revoke-keys"
Oct 30 20:09:32.935525 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:32.935507    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:32.967341 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:32.967304    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d5x6f\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:32.967433 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:32.967408    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqmv5\" (UniqueName: \"kubernetes.io/projected/9f87fd14-60d9-4835-b4e7-8c0661f076fa-kube-api-access-sqmv5\") pod \"maas-api-revoke-sub-2e1f5b5f-d5x6f\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:32.967485 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:32.967476    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d5x6f\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:32.987213 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:32.987184    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:09:33.036584 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:33.036561    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:09:33.038424 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:33.038406    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"]
Oct 30 20:09:33.068937 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:33.068914    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d5x6f\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:33.069032 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:33.068967    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d5x6f\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:33.069032 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:33.069022    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqmv5\" (UniqueName: \"kubernetes.io/projected/9f87fd14-60d9-4835-b4e7-8c0661f076fa-kube-api-access-sqmv5\") pod \"maas-api-revoke-sub-2e1f5b5f-d5x6f\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:33.069555 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:33.069540    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d5x6f\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:33.071167 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:33.071151    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d5x6f\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:33.125045 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:33.125017    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqmv5\" (UniqueName: \"kubernetes.io/projected/9f87fd14-60d9-4835-b4e7-8c0661f076fa-kube-api-access-sqmv5\") pod \"maas-api-revoke-sub-2e1f5b5f-d5x6f\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:33.248442 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:33.248396    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:33.622011 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:33.621985    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"]
Oct 30 20:09:33.624645 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:09:33.624621    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f87fd14_60d9_4835_b4e7_8c0661f076fa.slice/crio-5f96ee869a11f7fb30dbba3ade87c15378abf48642cd7016bcfbf462d7087e0c WatchSource:0}: Error finding container 5f96ee869a11f7fb30dbba3ade87c15378abf48642cd7016bcfbf462d7087e0c: Status 404 returned error can't find the container with id 5f96ee869a11f7fb30dbba3ade87c15378abf48642cd7016bcfbf462d7087e0c
Oct 30 20:09:33.718607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:33.718585    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f" event={"ID":"9f87fd14-60d9-4835-b4e7-8c0661f076fa","Type":"ContainerStarted","Data":"5f96ee869a11f7fb30dbba3ade87c15378abf48642cd7016bcfbf462d7087e0c"}
Oct 30 20:09:34.724096 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:34.724065    4420 generic.go:356] "Generic (PLEG): container finished" podID="9f87fd14-60d9-4835-b4e7-8c0661f076fa" containerID="a59cb30b9b45aad997bd0de16c644ac9306ba69067abf9cd96eeb20194151fcf" exitCode=0
Oct 30 20:09:34.724470 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:34.724146    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f" event={"ID":"9f87fd14-60d9-4835-b4e7-8c0661f076fa","Type":"ContainerDied","Data":"a59cb30b9b45aad997bd0de16c644ac9306ba69067abf9cd96eeb20194151fcf"}
Oct 30 20:09:35.194491 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:09:35.194456    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:09:35.863541 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:35.863518    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:35.895251 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:35.895229    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-xks-ca\") pod \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") "
Oct 30 20:09:35.895351 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:35.895281    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f87fd14-60d9-4835-b4e7-8c0661f076fa-kube-api-access-sqmv5\" (UniqueName: \"kubernetes.io/projected/9f87fd14-60d9-4835-b4e7-8c0661f076fa-kube-api-access-sqmv5\") pod \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") "
Oct 30 20:09:35.895351 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:35.895343    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-openshift-ca\") pod \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\" (UID: \"9f87fd14-60d9-4835-b4e7-8c0661f076fa\") "
Oct 30 20:09:35.895710 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:35.895683    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-openshift-ca" pod "9f87fd14-60d9-4835-b4e7-8c0661f076fa" (UID: "9f87fd14-60d9-4835-b4e7-8c0661f076fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:35.897264 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:35.897246    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-xks-ca" pod "9f87fd14-60d9-4835-b4e7-8c0661f076fa" (UID: "9f87fd14-60d9-4835-b4e7-8c0661f076fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:09:35.897336 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:35.897320    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f87fd14-60d9-4835-b4e7-8c0661f076fa-kube-api-access-sqmv5" pod "9f87fd14-60d9-4835-b4e7-8c0661f076fa" (UID: "9f87fd14-60d9-4835-b4e7-8c0661f076fa"). InnerVolumeSpecName "kube-api-access-sqmv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:35.997189 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:35.997106    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqmv5\" (UniqueName: \"kubernetes.io/projected/9f87fd14-60d9-4835-b4e7-8c0661f076fa-kube-api-access-sqmv5\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:35.997189 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:35.997138    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:35.997189 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:35.997156    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f87fd14-60d9-4835-b4e7-8c0661f076fa-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:36.733978 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:36.733956    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"
Oct 30 20:09:36.734126 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:36.733948    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f" event={"ID":"9f87fd14-60d9-4835-b4e7-8c0661f076fa","Type":"ContainerDied","Data":"5f96ee869a11f7fb30dbba3ade87c15378abf48642cd7016bcfbf462d7087e0c"}
Oct 30 20:09:36.734126 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:36.734106    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f96ee869a11f7fb30dbba3ade87c15378abf48642cd7016bcfbf462d7087e0c"
Oct 30 20:09:45.419012 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.418977    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"]
Oct 30 20:09:45.419573 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.419558    4420 cpu_manager.go:414] "containerMap: removing container" podUID="9f87fd14-60d9-4835-b4e7-8c0661f076fa" containerName="revoke-keys"
Oct 30 20:09:45.419615 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.419576    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f87fd14-60d9-4835-b4e7-8c0661f076fa" containerName="revoke-keys"
Oct 30 20:09:45.419671 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.419662    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f87fd14-60d9-4835-b4e7-8c0661f076fa" containerName="revoke-keys"
Oct 30 20:09:45.422801 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.422782    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:45.477072 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.477046    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:09:45.477584 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.477564    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln2bg\" (UniqueName: \"kubernetes.io/projected/1cd89073-2aad-45fe-9b7b-114985e4eb0b-kube-api-access-ln2bg\") pod \"maas-api-revoke-sub-ec843018-zd9wx\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:45.477706 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.477690    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zd9wx\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:45.477753 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.477722    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zd9wx\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:45.526324 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.526301    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:09:45.528443 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.528421    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"]
Oct 30 20:09:45.579132 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.579105    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zd9wx\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:45.579217 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.579142    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zd9wx\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:45.579298 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.579275    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2bg\" (UniqueName: \"kubernetes.io/projected/1cd89073-2aad-45fe-9b7b-114985e4eb0b-kube-api-access-ln2bg\") pod \"maas-api-revoke-sub-ec843018-zd9wx\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:45.579766 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.579746    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zd9wx\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:45.581333 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.581317    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zd9wx\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:45.635440 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.635417    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2bg\" (UniqueName: \"kubernetes.io/projected/1cd89073-2aad-45fe-9b7b-114985e4eb0b-kube-api-access-ln2bg\") pod \"maas-api-revoke-sub-ec843018-zd9wx\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:45.735069 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:45.734989    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:46.103403 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:46.103213    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"]
Oct 30 20:09:46.106223 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:09:46.106168    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd89073_2aad_45fe_9b7b_114985e4eb0b.slice/crio-50211b8f959c454e9fbe560b7812a1973bb8db88158106f668a693a1f15ef9bc WatchSource:0}: Error finding container 50211b8f959c454e9fbe560b7812a1973bb8db88158106f668a693a1f15ef9bc: Status 404 returned error can't find the container with id 50211b8f959c454e9fbe560b7812a1973bb8db88158106f668a693a1f15ef9bc
Oct 30 20:09:46.782970 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:46.782933    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx" event={"ID":"1cd89073-2aad-45fe-9b7b-114985e4eb0b","Type":"ContainerStarted","Data":"e71fd5a3a6341c974dbf6fcd27e4feb5ba12cf77ab1aa7def1f8e832cab2c03b"}
Oct 30 20:09:46.783305 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:46.782982    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx" event={"ID":"1cd89073-2aad-45fe-9b7b-114985e4eb0b","Type":"ContainerStarted","Data":"50211b8f959c454e9fbe560b7812a1973bb8db88158106f668a693a1f15ef9bc"}
Oct 30 20:09:46.897213 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:46.897096    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx" podStartSLOduration=1.897078555 podStartE2EDuration="1.897078555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:46.896011697 +0000 UTC m=+3234.107228841" watchObservedRunningTime="2026-10-30 20:09:46.897078555 +0000 UTC m=+3234.108295700"
Oct 30 20:09:47.789160 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:47.789127    4420 generic.go:356] "Generic (PLEG): container finished" podID="1cd89073-2aad-45fe-9b7b-114985e4eb0b" containerID="e71fd5a3a6341c974dbf6fcd27e4feb5ba12cf77ab1aa7def1f8e832cab2c03b" exitCode=0
Oct 30 20:09:47.789536 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:47.789191    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx" event={"ID":"1cd89073-2aad-45fe-9b7b-114985e4eb0b","Type":"ContainerDied","Data":"e71fd5a3a6341c974dbf6fcd27e4feb5ba12cf77ab1aa7def1f8e832cab2c03b"}
Oct 30 20:09:48.918855 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:48.918828    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:49.009193 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.009169    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-xks-ca\") pod \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") "
Oct 30 20:09:49.009320 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.009218    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-openshift-ca\") pod \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") "
Oct 30 20:09:49.009320 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.009251    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd89073-2aad-45fe-9b7b-114985e4eb0b-kube-api-access-ln2bg\" (UniqueName: \"kubernetes.io/projected/1cd89073-2aad-45fe-9b7b-114985e4eb0b-kube-api-access-ln2bg\") pod \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\" (UID: \"1cd89073-2aad-45fe-9b7b-114985e4eb0b\") "
Oct 30 20:09:49.009588 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.009556    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-openshift-ca" pod "1cd89073-2aad-45fe-9b7b-114985e4eb0b" (UID: "1cd89073-2aad-45fe-9b7b-114985e4eb0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:09:49.011363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.011325    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-xks-ca" pod "1cd89073-2aad-45fe-9b7b-114985e4eb0b" (UID: "1cd89073-2aad-45fe-9b7b-114985e4eb0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:09:49.011439 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.011388    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd89073-2aad-45fe-9b7b-114985e4eb0b-kube-api-access-ln2bg" pod "1cd89073-2aad-45fe-9b7b-114985e4eb0b" (UID: "1cd89073-2aad-45fe-9b7b-114985e4eb0b"). InnerVolumeSpecName "kube-api-access-ln2bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:09:49.110300 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.110247    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:49.110300 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.110280    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd89073-2aad-45fe-9b7b-114985e4eb0b-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:49.110300 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.110291    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln2bg\" (UniqueName: \"kubernetes.io/projected/1cd89073-2aad-45fe-9b7b-114985e4eb0b-kube-api-access-ln2bg\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:09:49.799102 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.799075    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"
Oct 30 20:09:49.799235 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.799121    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx" event={"ID":"1cd89073-2aad-45fe-9b7b-114985e4eb0b","Type":"ContainerDied","Data":"50211b8f959c454e9fbe560b7812a1973bb8db88158106f668a693a1f15ef9bc"}
Oct 30 20:09:49.799235 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:09:49.799169    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50211b8f959c454e9fbe560b7812a1973bb8db88158106f668a693a1f15ef9bc"
Oct 30 20:10:05.196008 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:10:05.195973    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:10:34.337418 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.337346    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"]
Oct 30 20:10:34.337995 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.337978    4420 cpu_manager.go:414] "containerMap: removing container" podUID="1cd89073-2aad-45fe-9b7b-114985e4eb0b" containerName="revoke-keys"
Oct 30 20:10:34.338075 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.337998    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd89073-2aad-45fe-9b7b-114985e4eb0b" containerName="revoke-keys"
Oct 30 20:10:34.338152 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.338140    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd89073-2aad-45fe-9b7b-114985e4eb0b" containerName="revoke-keys"
Oct 30 20:10:34.341435 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.341412    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:34.393080 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.393059    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:10:34.442789 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.442762    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:10:34.444984 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.444961    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"]
Oct 30 20:10:34.507404 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.507381    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-g22qj\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:34.507499 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.507421    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m5f7\" (UniqueName: \"kubernetes.io/projected/bac95397-9748-41ff-b3bb-132d415ec32d-kube-api-access-7m5f7\") pod \"maas-api-revoke-sub-046e1343-g22qj\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:34.507558 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.507545    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-g22qj\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:34.608803 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.608749    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-g22qj\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:34.608803 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.608792    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-g22qj\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:34.608988 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.608969    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m5f7\" (UniqueName: \"kubernetes.io/projected/bac95397-9748-41ff-b3bb-132d415ec32d-kube-api-access-7m5f7\") pod \"maas-api-revoke-sub-046e1343-g22qj\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:34.609334 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.609317    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-g22qj\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:34.611234 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.611209    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-g22qj\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:34.664658 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.664638    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m5f7\" (UniqueName: \"kubernetes.io/projected/bac95397-9748-41ff-b3bb-132d415ec32d-kube-api-access-7m5f7\") pod \"maas-api-revoke-sub-046e1343-g22qj\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:34.954782 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:34.954764    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:35.195435 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:10:35.195400    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:10:35.323567 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:35.323543    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"]
Oct 30 20:10:35.324984 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:10:35.324955    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac95397_9748_41ff_b3bb_132d415ec32d.slice/crio-8f2baa68b497109b0f3721c219c88d625d5d77eb4b693faed2ca17ff88f1d553 WatchSource:0}: Error finding container 8f2baa68b497109b0f3721c219c88d625d5d77eb4b693faed2ca17ff88f1d553: Status 404 returned error can't find the container with id 8f2baa68b497109b0f3721c219c88d625d5d77eb4b693faed2ca17ff88f1d553
Oct 30 20:10:36.037678 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:36.037648    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj" event={"ID":"bac95397-9748-41ff-b3bb-132d415ec32d","Type":"ContainerStarted","Data":"a027ce5b9e20ee6e17b6d083ab30820d1851fd4d5c42bd9c342463e7d591f01c"}
Oct 30 20:10:36.038096 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:36.037697    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj" event={"ID":"bac95397-9748-41ff-b3bb-132d415ec32d","Type":"ContainerStarted","Data":"8f2baa68b497109b0f3721c219c88d625d5d77eb4b693faed2ca17ff88f1d553"}
Oct 30 20:10:36.152311 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:36.152265    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj" podStartSLOduration=2.15225499 podStartE2EDuration="2.15225499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:36.151858271 +0000 UTC m=+3283.363075414" watchObservedRunningTime="2026-10-30 20:10:36.15225499 +0000 UTC m=+3283.363472133"
Oct 30 20:10:37.043777 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:37.043742    4420 generic.go:356] "Generic (PLEG): container finished" podID="bac95397-9748-41ff-b3bb-132d415ec32d" containerID="a027ce5b9e20ee6e17b6d083ab30820d1851fd4d5c42bd9c342463e7d591f01c" exitCode=0
Oct 30 20:10:37.044168 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:37.043792    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj" event={"ID":"bac95397-9748-41ff-b3bb-132d415ec32d","Type":"ContainerDied","Data":"a027ce5b9e20ee6e17b6d083ab30820d1851fd4d5c42bd9c342463e7d591f01c"}
Oct 30 20:10:38.173914 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:38.173892    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:38.340607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:38.340544    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-openshift-ca\") pod \"bac95397-9748-41ff-b3bb-132d415ec32d\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") "
Oct 30 20:10:38.340607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:38.340581    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac95397-9748-41ff-b3bb-132d415ec32d-kube-api-access-7m5f7\" (UniqueName: \"kubernetes.io/projected/bac95397-9748-41ff-b3bb-132d415ec32d-kube-api-access-7m5f7\") pod \"bac95397-9748-41ff-b3bb-132d415ec32d\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") "
Oct 30 20:10:38.340798 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:38.340615    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-xks-ca\") pod \"bac95397-9748-41ff-b3bb-132d415ec32d\" (UID: \"bac95397-9748-41ff-b3bb-132d415ec32d\") "
Oct 30 20:10:38.340971 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:38.340949    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-openshift-ca" pod "bac95397-9748-41ff-b3bb-132d415ec32d" (UID: "bac95397-9748-41ff-b3bb-132d415ec32d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:38.342776 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:38.342756    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-xks-ca" pod "bac95397-9748-41ff-b3bb-132d415ec32d" (UID: "bac95397-9748-41ff-b3bb-132d415ec32d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:38.342776 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:38.342761    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac95397-9748-41ff-b3bb-132d415ec32d-kube-api-access-7m5f7" pod "bac95397-9748-41ff-b3bb-132d415ec32d" (UID: "bac95397-9748-41ff-b3bb-132d415ec32d"). InnerVolumeSpecName "kube-api-access-7m5f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:38.441444 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:38.441424    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:10:38.441444 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:38.441444    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m5f7\" (UniqueName: \"kubernetes.io/projected/bac95397-9748-41ff-b3bb-132d415ec32d-kube-api-access-7m5f7\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:10:38.441568 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:38.441454    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bac95397-9748-41ff-b3bb-132d415ec32d-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:10:39.054414 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:39.054385    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"
Oct 30 20:10:39.054586 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:39.054388    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj" event={"ID":"bac95397-9748-41ff-b3bb-132d415ec32d","Type":"ContainerDied","Data":"8f2baa68b497109b0f3721c219c88d625d5d77eb4b693faed2ca17ff88f1d553"}
Oct 30 20:10:39.054586 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:39.054509    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f2baa68b497109b0f3721c219c88d625d5d77eb4b693faed2ca17ff88f1d553"
Oct 30 20:10:44.979414 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:44.979388    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"]
Oct 30 20:10:44.979898 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:44.979764    4420 cpu_manager.go:414] "containerMap: removing container" podUID="bac95397-9748-41ff-b3bb-132d415ec32d" containerName="revoke-keys"
Oct 30 20:10:44.979898 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:44.979774    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac95397-9748-41ff-b3bb-132d415ec32d" containerName="revoke-keys"
Oct 30 20:10:44.979898 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:44.979877    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac95397-9748-41ff-b3bb-132d415ec32d" containerName="revoke-keys"
Oct 30 20:10:44.983127 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:44.983109    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:45.034750 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.034720    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:10:45.084522 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.084491    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:10:45.088974 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.088947    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"]
Oct 30 20:10:45.094616 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.094589    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4t7bf\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:45.094789 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.094636    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4t7bf\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:45.094872 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.094803    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rbz\" (UniqueName: \"kubernetes.io/projected/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-kube-api-access-v8rbz\") pod \"maas-api-revoke-sub-d3fba056-4t7bf\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:45.195321 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.195300    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4t7bf\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:45.195435 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.195336    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4t7bf\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:45.195435 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.195394    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rbz\" (UniqueName: \"kubernetes.io/projected/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-kube-api-access-v8rbz\") pod \"maas-api-revoke-sub-d3fba056-4t7bf\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:45.196083 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.196061    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4t7bf\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:45.197625 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.197609    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4t7bf\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:45.251575 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.251529    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rbz\" (UniqueName: \"kubernetes.io/projected/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-kube-api-access-v8rbz\") pod \"maas-api-revoke-sub-d3fba056-4t7bf\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:45.295451 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.295427    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:45.665975 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:45.665881    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"]
Oct 30 20:10:45.668951 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:10:45.668924    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936c7002_06bb_4d0f_b7e0_9cae85e1f76c.slice/crio-3b77ba8b39aba82dcc8d38c2dff2463d32b13136557350341fde96fd063ffd68 WatchSource:0}: Error finding container 3b77ba8b39aba82dcc8d38c2dff2463d32b13136557350341fde96fd063ffd68: Status 404 returned error can't find the container with id 3b77ba8b39aba82dcc8d38c2dff2463d32b13136557350341fde96fd063ffd68
Oct 30 20:10:46.089447 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:46.089384    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf" event={"ID":"936c7002-06bb-4d0f-b7e0-9cae85e1f76c","Type":"ContainerStarted","Data":"2a44b8a1225a80dfc0d3e738c9ca78b085657f06450275286f9688c71b2c619a"}
Oct 30 20:10:46.089447 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:46.089435    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf" event={"ID":"936c7002-06bb-4d0f-b7e0-9cae85e1f76c","Type":"ContainerStarted","Data":"3b77ba8b39aba82dcc8d38c2dff2463d32b13136557350341fde96fd063ffd68"}
Oct 30 20:10:46.202094 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:46.202048    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf" podStartSLOduration=2.202033707 podStartE2EDuration="2.202033707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:46.200937805 +0000 UTC m=+3293.412154975" watchObservedRunningTime="2026-10-30 20:10:46.202033707 +0000 UTC m=+3293.413250908"
Oct 30 20:10:47.095422 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:47.095391    4420 generic.go:356] "Generic (PLEG): container finished" podID="936c7002-06bb-4d0f-b7e0-9cae85e1f76c" containerID="2a44b8a1225a80dfc0d3e738c9ca78b085657f06450275286f9688c71b2c619a" exitCode=0
Oct 30 20:10:47.095774 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:47.095427    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf" event={"ID":"936c7002-06bb-4d0f-b7e0-9cae85e1f76c","Type":"ContainerDied","Data":"2a44b8a1225a80dfc0d3e738c9ca78b085657f06450275286f9688c71b2c619a"}
Oct 30 20:10:48.432017 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:48.431777    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:10:48.437856 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:48.437280    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-xks-ca\") pod \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") "
Oct 30 20:10:48.437856 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:48.437360    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-openshift-ca\") pod \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") "
Oct 30 20:10:48.437856 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:48.437428    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-kube-api-access-v8rbz\" (UniqueName: \"kubernetes.io/projected/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-kube-api-access-v8rbz\") pod \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\" (UID: \"936c7002-06bb-4d0f-b7e0-9cae85e1f76c\") "
Oct 30 20:10:48.438189 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:48.438159    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-openshift-ca" pod "936c7002-06bb-4d0f-b7e0-9cae85e1f76c" (UID: "936c7002-06bb-4d0f-b7e0-9cae85e1f76c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:10:48.439931 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:48.439912    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-xks-ca" pod "936c7002-06bb-4d0f-b7e0-9cae85e1f76c" (UID: "936c7002-06bb-4d0f-b7e0-9cae85e1f76c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:10:48.439999 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:48.439968    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-kube-api-access-v8rbz" pod "936c7002-06bb-4d0f-b7e0-9cae85e1f76c" (UID: "936c7002-06bb-4d0f-b7e0-9cae85e1f76c"). InnerVolumeSpecName "kube-api-access-v8rbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:10:48.538610 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:48.538583    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8rbz\" (UniqueName: \"kubernetes.io/projected/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-kube-api-access-v8rbz\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:10:48.538724 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:48.538616    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:10:48.538724 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:48.538625    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936c7002-06bb-4d0f-b7e0-9cae85e1f76c-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:10:49.125446 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:49.125398    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf" event={"ID":"936c7002-06bb-4d0f-b7e0-9cae85e1f76c","Type":"ContainerDied","Data":"3b77ba8b39aba82dcc8d38c2dff2463d32b13136557350341fde96fd063ffd68"}
Oct 30 20:10:49.125723 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:49.125694    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b77ba8b39aba82dcc8d38c2dff2463d32b13136557350341fde96fd063ffd68"
Oct 30 20:10:49.125965 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:10:49.125950    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"
Oct 30 20:11:05.247000 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:11:05.246957    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:11:06.043968 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:06.043936    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-748lj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:11:06.044188 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:06.044004    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:11:06.516252 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:06.516219    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8r4gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:11:06.516748 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:06.516313    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" podUID="ca4ca4ee-283d-4613-8d97-09478295784d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:11:15.048962 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:15.048927    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"]
Oct 30 20:11:15.096727 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:15.096697    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e67581bd-2vn5c"]
Oct 30 20:11:16.894276 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:16.894239    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b4ee62b-dd50-4e0d-adfa-63403ae44527" path="/var/lib/kubelet/pods/8b4ee62b-dd50-4e0d-adfa-63403ae44527/volumes"
Oct 30 20:11:20.234729 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.234695    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"]
Oct 30 20:11:20.235209 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.235195    4420 cpu_manager.go:414] "containerMap: removing container" podUID="936c7002-06bb-4d0f-b7e0-9cae85e1f76c" containerName="revoke-keys"
Oct 30 20:11:20.235254 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.235210    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936c7002-06bb-4d0f-b7e0-9cae85e1f76c" containerName="revoke-keys"
Oct 30 20:11:20.235298 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.235289    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="936c7002-06bb-4d0f-b7e0-9cae85e1f76c" containerName="revoke-keys"
Oct 30 20:11:20.239443 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.239427    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:20.262715 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.262681    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-n78mp\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:20.262915 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.262883    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnqz\" (UniqueName: \"kubernetes.io/projected/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-kube-api-access-gtnqz\") pod \"maas-api-revoke-sub-ae594979-n78mp\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:20.262915 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.262912    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-n78mp\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:20.305741 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.300731    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"]
Oct 30 20:11:20.344552 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.344523    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:20.344552 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.344545    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:11:20.363960 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.363919    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnqz\" (UniqueName: \"kubernetes.io/projected/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-kube-api-access-gtnqz\") pod \"maas-api-revoke-sub-ae594979-n78mp\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:20.364146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.363978    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-n78mp\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:20.364146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.364011    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-n78mp\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:20.364699 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.364678    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-n78mp\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:20.366559 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.366535    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-n78mp\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:20.421710 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.421674    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnqz\" (UniqueName: \"kubernetes.io/projected/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-kube-api-access-gtnqz\") pod \"maas-api-revoke-sub-ae594979-n78mp\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:20.551779 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.551690    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:20.949293 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:20.949261    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"]
Oct 30 20:11:20.950864 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:11:20.950839    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d5f5c9_85a7_4b63_96b5_a769ff98cfd7.slice/crio-b3af929fdd93710f9b860e4fab6826721ce93c12b12692917ee94df9ebc090a0 WatchSource:0}: Error finding container b3af929fdd93710f9b860e4fab6826721ce93c12b12692917ee94df9ebc090a0: Status 404 returned error can't find the container with id b3af929fdd93710f9b860e4fab6826721ce93c12b12692917ee94df9ebc090a0
Oct 30 20:11:21.316924 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:21.316889    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp" event={"ID":"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7","Type":"ContainerStarted","Data":"a28552d75b5c966f898b727cf032d248c16f116f7bf517607c9c007b206f87f0"}
Oct 30 20:11:21.317332 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:21.316945    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp" event={"ID":"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7","Type":"ContainerStarted","Data":"b3af929fdd93710f9b860e4fab6826721ce93c12b12692917ee94df9ebc090a0"}
Oct 30 20:11:21.435626 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:21.435574    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp" podStartSLOduration=1.435557526 podStartE2EDuration="1.435557526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:21.433029564 +0000 UTC m=+3328.644246707" watchObservedRunningTime="2026-10-30 20:11:21.435557526 +0000 UTC m=+3328.646774670"
Oct 30 20:11:22.323239 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:22.323201    4420 generic.go:356] "Generic (PLEG): container finished" podID="70d5f5c9-85a7-4b63-96b5-a769ff98cfd7" containerID="a28552d75b5c966f898b727cf032d248c16f116f7bf517607c9c007b206f87f0" exitCode=0
Oct 30 20:11:22.323692 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:22.323268    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp" event={"ID":"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7","Type":"ContainerDied","Data":"a28552d75b5c966f898b727cf032d248c16f116f7bf517607c9c007b206f87f0"}
Oct 30 20:11:23.469501 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:23.469478    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:23.490828 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:23.490784    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-xks-ca\") pod \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") "
Oct 30 20:11:23.491147 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:23.490912    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-openshift-ca\") pod \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") "
Oct 30 20:11:23.491147 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:23.490966    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-kube-api-access-gtnqz\" (UniqueName: \"kubernetes.io/projected/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-kube-api-access-gtnqz\") pod \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\" (UID: \"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7\") "
Oct 30 20:11:23.491306 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:23.491287    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-openshift-ca" pod "70d5f5c9-85a7-4b63-96b5-a769ff98cfd7" (UID: "70d5f5c9-85a7-4b63-96b5-a769ff98cfd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:23.493097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:23.493075    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-xks-ca" pod "70d5f5c9-85a7-4b63-96b5-a769ff98cfd7" (UID: "70d5f5c9-85a7-4b63-96b5-a769ff98cfd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:23.493160 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:23.493138    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-kube-api-access-gtnqz" pod "70d5f5c9-85a7-4b63-96b5-a769ff98cfd7" (UID: "70d5f5c9-85a7-4b63-96b5-a769ff98cfd7"). InnerVolumeSpecName "kube-api-access-gtnqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:23.593054 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:23.592939    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:23.593054 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:23.592975    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:23.593054 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:23.592991    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtnqz\" (UniqueName: \"kubernetes.io/projected/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7-kube-api-access-gtnqz\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:24.334263 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:24.334228    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp" event={"ID":"70d5f5c9-85a7-4b63-96b5-a769ff98cfd7","Type":"ContainerDied","Data":"b3af929fdd93710f9b860e4fab6826721ce93c12b12692917ee94df9ebc090a0"}
Oct 30 20:11:24.334263 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:24.334248    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"
Oct 30 20:11:24.334479 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:24.334277    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3af929fdd93710f9b860e4fab6826721ce93c12b12692917ee94df9ebc090a0"
Oct 30 20:11:35.195171 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:11:35.195135    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:11:55.897416 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:55.897382    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"]
Oct 30 20:11:55.897856 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:55.897799    4420 cpu_manager.go:414] "containerMap: removing container" podUID="70d5f5c9-85a7-4b63-96b5-a769ff98cfd7" containerName="revoke-keys"
Oct 30 20:11:55.897856 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:55.897810    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d5f5c9-85a7-4b63-96b5-a769ff98cfd7" containerName="revoke-keys"
Oct 30 20:11:55.897929 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:55.897915    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d5f5c9-85a7-4b63-96b5-a769ff98cfd7" containerName="revoke-keys"
Oct 30 20:11:55.901213 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:55.901192    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:55.958422 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:55.958387    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"]
Oct 30 20:11:56.004659 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.004628    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:11:56.004659 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.004661    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:11:56.023720 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.023691    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz7cm\" (UniqueName: \"kubernetes.io/projected/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-kube-api-access-hz7cm\") pod \"maas-api-revoke-sub-a4aef7d7-2xbxn\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:56.023907 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.023749    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2xbxn\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:56.023907 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.023848    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2xbxn\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:56.124996 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.124960    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7cm\" (UniqueName: \"kubernetes.io/projected/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-kube-api-access-hz7cm\") pod \"maas-api-revoke-sub-a4aef7d7-2xbxn\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:56.125193 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.125023    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2xbxn\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:56.125193 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.125040    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2xbxn\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:56.125652 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.125632    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2xbxn\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:56.127478 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.127459    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-2xbxn\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:56.181802 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.181741    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz7cm\" (UniqueName: \"kubernetes.io/projected/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-kube-api-access-hz7cm\") pod \"maas-api-revoke-sub-a4aef7d7-2xbxn\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:56.213845 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.213794    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:56.586479 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:56.586452    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"]
Oct 30 20:11:56.588716 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:11:56.588686    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5893827_a4b0_41ef_9eab_e1dc37cfbe2a.slice/crio-ddbc56b7080cc89cbc90400a92b2555b250ed868d7044c370c38f8a8fec172fa WatchSource:0}: Error finding container ddbc56b7080cc89cbc90400a92b2555b250ed868d7044c370c38f8a8fec172fa: Status 404 returned error can't find the container with id ddbc56b7080cc89cbc90400a92b2555b250ed868d7044c370c38f8a8fec172fa
Oct 30 20:11:57.513081 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:57.513038    4420 generic.go:356] "Generic (PLEG): container finished" podID="e5893827-a4b0-41ef-9eab-e1dc37cfbe2a" containerID="456dd240d2e76d5301303ce1b2c6b45b8a5ac49cf028bddc4ef4036a6c7254bd" exitCode=0
Oct 30 20:11:57.513081 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:57.513081    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn" event={"ID":"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a","Type":"ContainerDied","Data":"456dd240d2e76d5301303ce1b2c6b45b8a5ac49cf028bddc4ef4036a6c7254bd"}
Oct 30 20:11:57.513508 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:57.513134    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn" event={"ID":"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a","Type":"ContainerStarted","Data":"ddbc56b7080cc89cbc90400a92b2555b250ed868d7044c370c38f8a8fec172fa"}
Oct 30 20:11:58.658525 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:58.658499    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:58.748210 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:58.748171    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-openshift-ca\") pod \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") "
Oct 30 20:11:58.748432 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:58.748261    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-kube-api-access-hz7cm\" (UniqueName: \"kubernetes.io/projected/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-kube-api-access-hz7cm\") pod \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") "
Oct 30 20:11:58.748432 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:58.748360    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-xks-ca\") pod \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\" (UID: \"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a\") "
Oct 30 20:11:58.748537 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:58.748511    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-openshift-ca" pod "e5893827-a4b0-41ef-9eab-e1dc37cfbe2a" (UID: "e5893827-a4b0-41ef-9eab-e1dc37cfbe2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:11:58.750537 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:58.750505    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-xks-ca" pod "e5893827-a4b0-41ef-9eab-e1dc37cfbe2a" (UID: "e5893827-a4b0-41ef-9eab-e1dc37cfbe2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:11:58.750537 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:58.750528    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-kube-api-access-hz7cm" pod "e5893827-a4b0-41ef-9eab-e1dc37cfbe2a" (UID: "e5893827-a4b0-41ef-9eab-e1dc37cfbe2a"). InnerVolumeSpecName "kube-api-access-hz7cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:11:58.849661 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:58.849571    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:58.849661 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:58.849605    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz7cm\" (UniqueName: \"kubernetes.io/projected/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-kube-api-access-hz7cm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:58.849661 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:58.849615    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:11:59.523699 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:59.523671    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"
Oct 30 20:11:59.523922 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:59.523670    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn" event={"ID":"e5893827-a4b0-41ef-9eab-e1dc37cfbe2a","Type":"ContainerDied","Data":"ddbc56b7080cc89cbc90400a92b2555b250ed868d7044c370c38f8a8fec172fa"}
Oct 30 20:11:59.523922 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:11:59.523807    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddbc56b7080cc89cbc90400a92b2555b250ed868d7044c370c38f8a8fec172fa"
Oct 30 20:12:05.195701 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:12:05.195613    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:12:20.048691 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:20.048656    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"]
Oct 30 20:12:20.098775 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:20.098741    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-fqjwt"]
Oct 30 20:12:20.894461 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:20.894429    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c788fa3f-ff2c-4756-9fc0-de975d013aa3" path="/var/lib/kubelet/pods/c788fa3f-ff2c-4756-9fc0-de975d013aa3/volumes"
Oct 30 20:12:35.196927 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:12:35.196882    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:12:39.989240 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:39.989198    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"]
Oct 30 20:12:39.989859 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:39.989840    4420 cpu_manager.go:414] "containerMap: removing container" podUID="e5893827-a4b0-41ef-9eab-e1dc37cfbe2a" containerName="revoke-keys"
Oct 30 20:12:39.989960 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:39.989861    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5893827-a4b0-41ef-9eab-e1dc37cfbe2a" containerName="revoke-keys"
Oct 30 20:12:39.990018 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:39.989996    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5893827-a4b0-41ef-9eab-e1dc37cfbe2a" containerName="revoke-keys"
Oct 30 20:12:39.993371 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:39.993351    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:40.074588 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.074548    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9fml\" (UniqueName: \"kubernetes.io/projected/fbf71c7d-5a29-498e-bf2e-0351c92092c8-kube-api-access-g9fml\") pod \"maas-api-revoke-sub-79805ca9-xdvbz\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:40.074778 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.074593    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xdvbz\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:40.074778 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.074745    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xdvbz\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:40.075618 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.075579    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:12:40.078009 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.077989    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"]
Oct 30 20:12:40.094805 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.094778    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:12:40.175465 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.175428    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9fml\" (UniqueName: \"kubernetes.io/projected/fbf71c7d-5a29-498e-bf2e-0351c92092c8-kube-api-access-g9fml\") pod \"maas-api-revoke-sub-79805ca9-xdvbz\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:40.175465 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.175469    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xdvbz\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:40.175699 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.175508    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xdvbz\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:40.176213 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.176190    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xdvbz\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:40.177951 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.177935    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xdvbz\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:40.231852 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.231787    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9fml\" (UniqueName: \"kubernetes.io/projected/fbf71c7d-5a29-498e-bf2e-0351c92092c8-kube-api-access-g9fml\") pod \"maas-api-revoke-sub-79805ca9-xdvbz\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:40.306887 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.306775    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:40.681865 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.681840    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"]
Oct 30 20:12:40.684032 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:12:40.683999    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf71c7d_5a29_498e_bf2e_0351c92092c8.slice/crio-4f75e06bbad6c64114bf3ebb4eb51a93ae655713653f49d6f907e19a764a895a WatchSource:0}: Error finding container 4f75e06bbad6c64114bf3ebb4eb51a93ae655713653f49d6f907e19a764a895a: Status 404 returned error can't find the container with id 4f75e06bbad6c64114bf3ebb4eb51a93ae655713653f49d6f907e19a764a895a
Oct 30 20:12:40.734524 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:40.734485    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz" event={"ID":"fbf71c7d-5a29-498e-bf2e-0351c92092c8","Type":"ContainerStarted","Data":"4f75e06bbad6c64114bf3ebb4eb51a93ae655713653f49d6f907e19a764a895a"}
Oct 30 20:12:41.741744 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:41.741710    4420 generic.go:356] "Generic (PLEG): container finished" podID="fbf71c7d-5a29-498e-bf2e-0351c92092c8" containerID="5897912d0b39d5ff2d5a139de30f7f0a35025af91e95613369743c4e5c16c919" exitCode=0
Oct 30 20:12:41.742158 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:41.741795    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz" event={"ID":"fbf71c7d-5a29-498e-bf2e-0351c92092c8","Type":"ContainerDied","Data":"5897912d0b39d5ff2d5a139de30f7f0a35025af91e95613369743c4e5c16c919"}
Oct 30 20:12:42.882036 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:42.882007    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:43.002175 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.002080    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf71c7d-5a29-498e-bf2e-0351c92092c8-kube-api-access-g9fml\" (UniqueName: \"kubernetes.io/projected/fbf71c7d-5a29-498e-bf2e-0351c92092c8-kube-api-access-g9fml\") pod \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") "
Oct 30 20:12:43.002356 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.002278    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-xks-ca\") pod \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") "
Oct 30 20:12:43.002356 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.002341    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-openshift-ca\") pod \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\" (UID: \"fbf71c7d-5a29-498e-bf2e-0351c92092c8\") "
Oct 30 20:12:43.002803 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.002774    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-openshift-ca" pod "fbf71c7d-5a29-498e-bf2e-0351c92092c8" (UID: "fbf71c7d-5a29-498e-bf2e-0351c92092c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:12:43.004582 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.004556    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-xks-ca" pod "fbf71c7d-5a29-498e-bf2e-0351c92092c8" (UID: "fbf71c7d-5a29-498e-bf2e-0351c92092c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:12:43.004725 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.004712    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf71c7d-5a29-498e-bf2e-0351c92092c8-kube-api-access-g9fml" pod "fbf71c7d-5a29-498e-bf2e-0351c92092c8" (UID: "fbf71c7d-5a29-498e-bf2e-0351c92092c8"). InnerVolumeSpecName "kube-api-access-g9fml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:12:43.104270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.104226    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.104270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.104263    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf71c7d-5a29-498e-bf2e-0351c92092c8-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.104270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.104274    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9fml\" (UniqueName: \"kubernetes.io/projected/fbf71c7d-5a29-498e-bf2e-0351c92092c8-kube-api-access-g9fml\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:12:43.752499 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.752470    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"
Oct 30 20:12:43.752658 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.752461    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz" event={"ID":"fbf71c7d-5a29-498e-bf2e-0351c92092c8","Type":"ContainerDied","Data":"4f75e06bbad6c64114bf3ebb4eb51a93ae655713653f49d6f907e19a764a895a"}
Oct 30 20:12:43.752658 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:43.752593    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f75e06bbad6c64114bf3ebb4eb51a93ae655713653f49d6f907e19a764a895a"
Oct 30 20:12:53.836063 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:53.836031    4420 scope.go:122] "RemoveContainer" containerID="8fa069e264f7946f97308e2028df7f9a66fadc35bb422595a6b98320a06aede5"
Oct 30 20:12:53.844801 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:12:53.844775    4420 scope.go:122] "RemoveContainer" containerID="684207f110fda394faffa327f781d0b81f8632c6197bb3be1e63804533d126f7"
Oct 30 20:13:04.316714 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.316682    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"]
Oct 30 20:13:04.317179 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.317164    4420 cpu_manager.go:414] "containerMap: removing container" podUID="fbf71c7d-5a29-498e-bf2e-0351c92092c8" containerName="revoke-keys"
Oct 30 20:13:04.317253 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.317180    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf71c7d-5a29-498e-bf2e-0351c92092c8" containerName="revoke-keys"
Oct 30 20:13:04.317312 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.317255    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf71c7d-5a29-498e-bf2e-0351c92092c8" containerName="revoke-keys"
Oct 30 20:13:04.320632 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.320615    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:04.414473 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.414438    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-stvpc\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:04.414635 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.414501    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb47l\" (UniqueName: \"kubernetes.io/projected/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-kube-api-access-nb47l\") pod \"maas-api-revoke-sub-0b7e8edb-stvpc\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:04.414673 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.414632    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-stvpc\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:04.423957 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.423939    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:13:04.424057 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.423975    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:13:04.426339 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.426320    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"]
Oct 30 20:13:04.515418 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.515394    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-stvpc\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:04.515555 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.515448    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb47l\" (UniqueName: \"kubernetes.io/projected/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-kube-api-access-nb47l\") pod \"maas-api-revoke-sub-0b7e8edb-stvpc\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:04.515555 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.515488    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-stvpc\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:04.516043 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.516014    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-stvpc\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:04.517753 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.517734    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-stvpc\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:04.572123 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.572065    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb47l\" (UniqueName: \"kubernetes.io/projected/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-kube-api-access-nb47l\") pod \"maas-api-revoke-sub-0b7e8edb-stvpc\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:04.633663 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:04.633636    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:05.005148 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:05.004971    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"]
Oct 30 20:13:05.007401 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:13:05.007378    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efd55bf_0c06_4e3f_8d5f_7c9ad92d3bd1.slice/crio-28ac782157666b4b695b0fa54a43c1894e891ab4cad82025b0e76fb8e0256c56 WatchSource:0}: Error finding container 28ac782157666b4b695b0fa54a43c1894e891ab4cad82025b0e76fb8e0256c56: Status 404 returned error can't find the container with id 28ac782157666b4b695b0fa54a43c1894e891ab4cad82025b0e76fb8e0256c56
Oct 30 20:13:05.195440 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:13:05.195409    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:13:05.882994 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:05.882958    4420 generic.go:356] "Generic (PLEG): container finished" podID="2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1" containerID="0ed02c317a34a2b0bbf80179d0191fc748abef5c511f4d231f73dafaedf0b8cc" exitCode=0
Oct 30 20:13:05.883383 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:05.883045    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc" event={"ID":"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1","Type":"ContainerDied","Data":"0ed02c317a34a2b0bbf80179d0191fc748abef5c511f4d231f73dafaedf0b8cc"}
Oct 30 20:13:05.883383 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:05.883089    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc" event={"ID":"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1","Type":"ContainerStarted","Data":"28ac782157666b4b695b0fa54a43c1894e891ab4cad82025b0e76fb8e0256c56"}
Oct 30 20:13:07.023389 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.023366    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:07.141420 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.141351    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-openshift-ca\") pod \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") "
Oct 30 20:13:07.141420 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.141416    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-xks-ca\") pod \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") "
Oct 30 20:13:07.141641 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.141527    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-kube-api-access-nb47l\" (UniqueName: \"kubernetes.io/projected/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-kube-api-access-nb47l\") pod \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\" (UID: \"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1\") "
Oct 30 20:13:07.141772 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.141751    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-openshift-ca" pod "2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1" (UID: "2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:07.141876 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.141862    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:13:07.143701 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.143677    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-kube-api-access-nb47l" pod "2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1" (UID: "2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1"). InnerVolumeSpecName "kube-api-access-nb47l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:07.143923 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.143904    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-xks-ca" pod "2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1" (UID: "2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:07.243289 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.243255    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb47l\" (UniqueName: \"kubernetes.io/projected/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-kube-api-access-nb47l\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:13:07.243289 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.243287    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:13:07.897429 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.897394    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc" event={"ID":"2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1","Type":"ContainerDied","Data":"28ac782157666b4b695b0fa54a43c1894e891ab4cad82025b0e76fb8e0256c56"}
Oct 30 20:13:07.897604 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.897438    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ac782157666b4b695b0fa54a43c1894e891ab4cad82025b0e76fb8e0256c56"
Oct 30 20:13:07.897604 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:07.897405    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"
Oct 30 20:13:15.607747 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.607708    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"]
Oct 30 20:13:15.608399 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.608381    4420 cpu_manager.go:414] "containerMap: removing container" podUID="2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1" containerName="revoke-keys"
Oct 30 20:13:15.608513 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.608403    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1" containerName="revoke-keys"
Oct 30 20:13:15.608571 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.608526    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1" containerName="revoke-keys"
Oct 30 20:13:15.611859 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.611839    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:15.621667 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.621639    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gshg\" (UniqueName: \"kubernetes.io/projected/d11947b1-5f34-41e1-8aa7-372fcf4a49af-kube-api-access-7gshg\") pod \"maas-api-revoke-sub-3d4156f1-w66pz\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:15.621789 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.621681    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w66pz\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:15.621789 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.621713    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w66pz\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:15.669451 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.669420    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"]
Oct 30 20:13:15.715066 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.715042    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:13:15.715066 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.715043    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:13:15.722574 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.722544    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gshg\" (UniqueName: \"kubernetes.io/projected/d11947b1-5f34-41e1-8aa7-372fcf4a49af-kube-api-access-7gshg\") pod \"maas-api-revoke-sub-3d4156f1-w66pz\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:15.722696 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.722591    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w66pz\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:15.722696 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.722630    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w66pz\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:15.723299 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.723281    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w66pz\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:15.725148 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.725121    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w66pz\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:15.778593 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.778573    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gshg\" (UniqueName: \"kubernetes.io/projected/d11947b1-5f34-41e1-8aa7-372fcf4a49af-kube-api-access-7gshg\") pod \"maas-api-revoke-sub-3d4156f1-w66pz\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:15.925386 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:15.925354    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:16.301010 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:16.300981    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"]
Oct 30 20:13:16.302429 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:13:16.302402    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11947b1_5f34_41e1_8aa7_372fcf4a49af.slice/crio-06232bec19287c12a4643b45b9d1b3d865dcd620115970fe2a20b00b03fed343 WatchSource:0}: Error finding container 06232bec19287c12a4643b45b9d1b3d865dcd620115970fe2a20b00b03fed343: Status 404 returned error can't find the container with id 06232bec19287c12a4643b45b9d1b3d865dcd620115970fe2a20b00b03fed343
Oct 30 20:13:16.940705 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:16.940667    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz" event={"ID":"d11947b1-5f34-41e1-8aa7-372fcf4a49af","Type":"ContainerStarted","Data":"4bf58335dbf8ba8a7ff37fe6f4de5c98eb3eebb40c0a8a3337df47cfa89e733b"}
Oct 30 20:13:16.941127 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:16.940720    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz" event={"ID":"d11947b1-5f34-41e1-8aa7-372fcf4a49af","Type":"ContainerStarted","Data":"06232bec19287c12a4643b45b9d1b3d865dcd620115970fe2a20b00b03fed343"}
Oct 30 20:13:17.054637 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:17.054590    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz" podStartSLOduration=2.054576385 podStartE2EDuration="2.054576385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:17.053854518 +0000 UTC m=+3444.265071663" watchObservedRunningTime="2026-10-30 20:13:17.054576385 +0000 UTC m=+3444.265793528"
Oct 30 20:13:17.947741 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:17.947707    4420 generic.go:356] "Generic (PLEG): container finished" podID="d11947b1-5f34-41e1-8aa7-372fcf4a49af" containerID="4bf58335dbf8ba8a7ff37fe6f4de5c98eb3eebb40c0a8a3337df47cfa89e733b" exitCode=0
Oct 30 20:13:17.948181 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:17.947754    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz" event={"ID":"d11947b1-5f34-41e1-8aa7-372fcf4a49af","Type":"ContainerDied","Data":"4bf58335dbf8ba8a7ff37fe6f4de5c98eb3eebb40c0a8a3337df47cfa89e733b"}
Oct 30 20:13:19.161748 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.161715    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:19.259984 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.259900    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-xks-ca\") pod \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") "
Oct 30 20:13:19.260158 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.259989    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11947b1-5f34-41e1-8aa7-372fcf4a49af-kube-api-access-7gshg\" (UniqueName: \"kubernetes.io/projected/d11947b1-5f34-41e1-8aa7-372fcf4a49af-kube-api-access-7gshg\") pod \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") "
Oct 30 20:13:19.260158 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.260027    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-openshift-ca\") pod \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\" (UID: \"d11947b1-5f34-41e1-8aa7-372fcf4a49af\") "
Oct 30 20:13:19.260454 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.260431    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-openshift-ca" pod "d11947b1-5f34-41e1-8aa7-372fcf4a49af" (UID: "d11947b1-5f34-41e1-8aa7-372fcf4a49af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:19.262270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.262243    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-xks-ca" pod "d11947b1-5f34-41e1-8aa7-372fcf4a49af" (UID: "d11947b1-5f34-41e1-8aa7-372fcf4a49af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:19.262270 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.262251    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11947b1-5f34-41e1-8aa7-372fcf4a49af-kube-api-access-7gshg" pod "d11947b1-5f34-41e1-8aa7-372fcf4a49af" (UID: "d11947b1-5f34-41e1-8aa7-372fcf4a49af"). InnerVolumeSpecName "kube-api-access-7gshg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:19.361352 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.361311    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gshg\" (UniqueName: \"kubernetes.io/projected/d11947b1-5f34-41e1-8aa7-372fcf4a49af-kube-api-access-7gshg\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:13:19.361352 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.361354    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:13:19.361352 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.361369    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11947b1-5f34-41e1-8aa7-372fcf4a49af-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:13:19.964760 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.964730    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"
Oct 30 20:13:19.964760 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.964730    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz" event={"ID":"d11947b1-5f34-41e1-8aa7-372fcf4a49af","Type":"ContainerDied","Data":"06232bec19287c12a4643b45b9d1b3d865dcd620115970fe2a20b00b03fed343"}
Oct 30 20:13:19.964992 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:19.964837    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06232bec19287c12a4643b45b9d1b3d865dcd620115970fe2a20b00b03fed343"
Oct 30 20:13:35.195685 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:13:35.195593    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:13:46.210524 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.210437    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"]
Oct 30 20:13:46.213349 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.213321    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w66pz"]
Oct 30 20:13:46.330118 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.330089    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"]
Oct 30 20:13:46.330530 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.330517    4420 cpu_manager.go:414] "containerMap: removing container" podUID="d11947b1-5f34-41e1-8aa7-372fcf4a49af" containerName="revoke-keys"
Oct 30 20:13:46.330584 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.330532    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11947b1-5f34-41e1-8aa7-372fcf4a49af" containerName="revoke-keys"
Oct 30 20:13:46.330625 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.330606    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11947b1-5f34-41e1-8aa7-372fcf4a49af" containerName="revoke-keys"
Oct 30 20:13:46.333845 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.333797    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:46.388252 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.388227    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:13:46.388883 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.388862    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:13:46.392056 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.392034    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"]
Oct 30 20:13:46.448474 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.448441    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw7kl\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:46.448624 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.448500    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-kube-api-access-jkf8d\") pod \"maas-api-revoke-sub-3d4156f1-bw7kl\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:46.448705 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.448682    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw7kl\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:46.549755 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.549675    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw7kl\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:46.549755 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.549736    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-kube-api-access-jkf8d\") pod \"maas-api-revoke-sub-3d4156f1-bw7kl\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:46.550026 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.549800    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw7kl\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:46.550492 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.550469    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw7kl\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:46.552097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.552076    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bw7kl\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:46.606325 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.606306    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-kube-api-access-jkf8d\") pod \"maas-api-revoke-sub-3d4156f1-bw7kl\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:46.647654 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.647631    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:46.898694 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:46.895497    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11947b1-5f34-41e1-8aa7-372fcf4a49af" path="/var/lib/kubelet/pods/d11947b1-5f34-41e1-8aa7-372fcf4a49af/volumes"
Oct 30 20:13:47.019257 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:47.019226    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"]
Oct 30 20:13:47.019691 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:13:47.019667    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb710c03f_1d8f_4299_9ee3_5c5a4b573a1c.slice/crio-9490c1559963d7663eabbc96198e6c2a6c8eebc4ffd0a8b70469ff73fbbf26ba WatchSource:0}: Error finding container 9490c1559963d7663eabbc96198e6c2a6c8eebc4ffd0a8b70469ff73fbbf26ba: Status 404 returned error can't find the container with id 9490c1559963d7663eabbc96198e6c2a6c8eebc4ffd0a8b70469ff73fbbf26ba
Oct 30 20:13:47.106451 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:47.106419    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl" event={"ID":"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c","Type":"ContainerStarted","Data":"9490c1559963d7663eabbc96198e6c2a6c8eebc4ffd0a8b70469ff73fbbf26ba"}
Oct 30 20:13:48.113146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:48.113069    4420 generic.go:356] "Generic (PLEG): container finished" podID="b710c03f-1d8f-4299-9ee3-5c5a4b573a1c" containerID="073934bd40fc2e16be02b24c15f2b05192af930151754536c46c7a55cc2bdd9f" exitCode=0
Oct 30 20:13:48.113146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:48.113133    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl" event={"ID":"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c","Type":"ContainerDied","Data":"073934bd40fc2e16be02b24c15f2b05192af930151754536c46c7a55cc2bdd9f"}
Oct 30 20:13:49.255172 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:49.255149    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:49.274237 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:49.274215    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-xks-ca\") pod \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") "
Oct 30 20:13:49.274376 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:49.274306    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-kube-api-access-jkf8d\") pod \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") "
Oct 30 20:13:49.274376 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:49.274339    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-openshift-ca\") pod \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\" (UID: \"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c\") "
Oct 30 20:13:49.274712 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:49.274692    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-openshift-ca" pod "b710c03f-1d8f-4299-9ee3-5c5a4b573a1c" (UID: "b710c03f-1d8f-4299-9ee3-5c5a4b573a1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:13:49.276245 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:49.276225    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-xks-ca" pod "b710c03f-1d8f-4299-9ee3-5c5a4b573a1c" (UID: "b710c03f-1d8f-4299-9ee3-5c5a4b573a1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:13:49.276548 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:49.276530    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-kube-api-access-jkf8d" pod "b710c03f-1d8f-4299-9ee3-5c5a4b573a1c" (UID: "b710c03f-1d8f-4299-9ee3-5c5a4b573a1c"). InnerVolumeSpecName "kube-api-access-jkf8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:13:49.375474 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:49.375422    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:13:49.375474 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:49.375445    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkf8d\" (UniqueName: \"kubernetes.io/projected/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-kube-api-access-jkf8d\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:13:49.375474 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:49.375456    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:13:50.124087 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:50.124056    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl" event={"ID":"b710c03f-1d8f-4299-9ee3-5c5a4b573a1c","Type":"ContainerDied","Data":"9490c1559963d7663eabbc96198e6c2a6c8eebc4ffd0a8b70469ff73fbbf26ba"}
Oct 30 20:13:50.124250 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:50.124092    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"
Oct 30 20:13:50.124250 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:50.124100    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9490c1559963d7663eabbc96198e6c2a6c8eebc4ffd0a8b70469ff73fbbf26ba"
Oct 30 20:13:51.047781 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:51.047752    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"]
Oct 30 20:13:51.068643 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:51.068614    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-87k75"]
Oct 30 20:13:52.894839 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:52.894791    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a350d00b-cf17-417f-b0d8-dd29dd979f51" path="/var/lib/kubelet/pods/a350d00b-cf17-417f-b0d8-dd29dd979f51/volumes"
Oct 30 20:13:53.867309 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:13:53.867281    4420 scope.go:122] "RemoveContainer" containerID="a7dd354eff61a2623056a3594c4d0da2afd82fc3db4b75408cb7d8d2d4e23b23"
Oct 30 20:14:05.195861 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:14:05.195809    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:14:30.558045 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.558004    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"]
Oct 30 20:14:30.599418 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.599391    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bw7kl"]
Oct 30 20:14:30.666632 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.666605    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"]
Oct 30 20:14:30.667074 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.667061    4420 cpu_manager.go:414] "containerMap: removing container" podUID="b710c03f-1d8f-4299-9ee3-5c5a4b573a1c" containerName="revoke-keys"
Oct 30 20:14:30.667123 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.667076    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b710c03f-1d8f-4299-9ee3-5c5a4b573a1c" containerName="revoke-keys"
Oct 30 20:14:30.667170 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.667161    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="b710c03f-1d8f-4299-9ee3-5c5a4b573a1c" containerName="revoke-keys"
Oct 30 20:14:30.670318 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.670303    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:30.724731 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.724683    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:14:30.724958 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.724945    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:14:30.727769 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.727747    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"]
Oct 30 20:14:30.824083 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.824018    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xbt\" (UniqueName: \"kubernetes.io/projected/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-kube-api-access-w7xbt\") pod \"maas-api-revoke-sub-3d4156f1-n46vj\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:30.824202 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.824092    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n46vj\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:30.824202 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.824170    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n46vj\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:30.894600 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.894572    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b710c03f-1d8f-4299-9ee3-5c5a4b573a1c" path="/var/lib/kubelet/pods/b710c03f-1d8f-4299-9ee3-5c5a4b573a1c/volumes"
Oct 30 20:14:30.924771 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.924743    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xbt\" (UniqueName: \"kubernetes.io/projected/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-kube-api-access-w7xbt\") pod \"maas-api-revoke-sub-3d4156f1-n46vj\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:30.924926 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.924798    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n46vj\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:30.924926 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.924854    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n46vj\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:30.925502 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.925486    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-n46vj\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:30.927286 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.927270    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-n46vj\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:30.980653 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.980625    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xbt\" (UniqueName: \"kubernetes.io/projected/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-kube-api-access-w7xbt\") pod \"maas-api-revoke-sub-3d4156f1-n46vj\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:30.983426 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:30.983412    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:31.353426 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:31.353396    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"]
Oct 30 20:14:31.355567 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:14:31.355523    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4c6289_6c5a_45d4_a7ed_ce8338b8c5a9.slice/crio-f4c0c012f797b36d5b65807e1e71c85a49a463dc85b4a733ef862624f439d9fa WatchSource:0}: Error finding container f4c0c012f797b36d5b65807e1e71c85a49a463dc85b4a733ef862624f439d9fa: Status 404 returned error can't find the container with id f4c0c012f797b36d5b65807e1e71c85a49a463dc85b4a733ef862624f439d9fa
Oct 30 20:14:32.332681 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:32.332645    4420 generic.go:356] "Generic (PLEG): container finished" podID="2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9" containerID="bd36dfb8c0286bea428803abdceea1a85c0bfb132bdb95768c28e1067db180a6" exitCode=0
Oct 30 20:14:32.333030 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:32.332700    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj" event={"ID":"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9","Type":"ContainerDied","Data":"bd36dfb8c0286bea428803abdceea1a85c0bfb132bdb95768c28e1067db180a6"}
Oct 30 20:14:32.333030 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:32.332745    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj" event={"ID":"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9","Type":"ContainerStarted","Data":"f4c0c012f797b36d5b65807e1e71c85a49a463dc85b4a733ef862624f439d9fa"}
Oct 30 20:14:33.464183 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:33.464161    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:33.545993 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:33.545969    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-openshift-ca\") pod \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") "
Oct 30 20:14:33.546109 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:33.546023    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-xks-ca\") pod \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") "
Oct 30 20:14:33.546183 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:33.546122    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-kube-api-access-w7xbt\" (UniqueName: \"kubernetes.io/projected/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-kube-api-access-w7xbt\") pod \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\" (UID: \"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9\") "
Oct 30 20:14:33.546309 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:33.546287    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-openshift-ca" pod "2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9" (UID: "2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:14:33.548122 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:33.548100    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-kube-api-access-w7xbt" pod "2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9" (UID: "2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9"). InnerVolumeSpecName "kube-api-access-w7xbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:14:33.548225 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:33.548137    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-xks-ca" pod "2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9" (UID: "2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:14:33.647220 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:33.647197    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:33.647318 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:33.647220    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:33.647318 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:33.647230    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7xbt\" (UniqueName: \"kubernetes.io/projected/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9-kube-api-access-w7xbt\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:14:34.343335 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:34.343308    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"
Oct 30 20:14:34.343511 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:34.343350    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj" event={"ID":"2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9","Type":"ContainerDied","Data":"f4c0c012f797b36d5b65807e1e71c85a49a463dc85b4a733ef862624f439d9fa"}
Oct 30 20:14:34.343511 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:34.343401    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4c0c012f797b36d5b65807e1e71c85a49a463dc85b4a733ef862624f439d9fa"
Oct 30 20:14:35.196223 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:14:35.196182    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:14:36.046544 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:36.046513    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"]
Oct 30 20:14:36.095452 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:36.095427    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d5x6f"]
Oct 30 20:14:36.893834 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:36.893784    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f87fd14-60d9-4835-b4e7-8c0661f076fa" path="/var/lib/kubelet/pods/9f87fd14-60d9-4835-b4e7-8c0661f076fa/volumes"
Oct 30 20:14:49.048037 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:49.048001    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"]
Oct 30 20:14:49.094582 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:49.094548    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zd9wx"]
Oct 30 20:14:50.894444 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:50.894409    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd89073-2aad-45fe-9b7b-114985e4eb0b" path="/var/lib/kubelet/pods/1cd89073-2aad-45fe-9b7b-114985e4eb0b/volumes"
Oct 30 20:14:53.887203 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:14:53.887173    4420 scope.go:122] "RemoveContainer" containerID="a1f5c2c359f0551f8e4ea2ae1021d12c0b18e15267ce9eaab2ad3d7bea485aeb"
Oct 30 20:15:00.164663 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.164623    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"]
Oct 30 20:15:00.165306 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.165287    4420 cpu_manager.go:414] "containerMap: removing container" podUID="2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9" containerName="revoke-keys"
Oct 30 20:15:00.165381 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.165307    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9" containerName="revoke-keys"
Oct 30 20:15:00.165437 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.165396    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9" containerName="revoke-keys"
Oct 30 20:15:00.168513 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.168494    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"
Oct 30 20:15:00.227364 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.227327    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"]
Oct 30 20:15:00.270431 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.270397    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:15:00.271859 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.271838    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6857f\" (UniqueName: \"kubernetes.io/projected/0acab457-5f11-4d97-9467-1e86f8d6f99e-kube-api-access-6857f\") pod \"maas-api-key-cleanup-29889855-sffmg\" (UID: \"0acab457-5f11-4d97-9467-1e86f8d6f99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"
Oct 30 20:15:00.272014 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.272000    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acab457-5f11-4d97-9467-1e86f8d6f99e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-sffmg\" (UID: \"0acab457-5f11-4d97-9467-1e86f8d6f99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"
Oct 30 20:15:00.323144 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.323109    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"]
Oct 30 20:15:00.327586 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.327560    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.356526 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.356488    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"]
Oct 30 20:15:00.372859 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.372834    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-socket\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.373006 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.372872    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-token\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.373006 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.372903    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istiod-ca-cert\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.373006 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.372999    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-data\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.373181 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.373024    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-envoy\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.373181 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.373131    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlg7\" (UniqueName: \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-kube-api-access-bzlg7\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.373263 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.373184    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acab457-5f11-4d97-9467-1e86f8d6f99e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-sffmg\" (UID: \"0acab457-5f11-4d97-9467-1e86f8d6f99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"
Oct 30 20:15:00.373263 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.373244    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-credential-socket\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.373342 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.373274    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-certs\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.373342 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.373295    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-podinfo\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.373342 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.373330    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6857f\" (UniqueName: \"kubernetes.io/projected/0acab457-5f11-4d97-9467-1e86f8d6f99e-kube-api-access-6857f\") pod \"maas-api-key-cleanup-29889855-sffmg\" (UID: \"0acab457-5f11-4d97-9467-1e86f8d6f99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"
Oct 30 20:15:00.373879 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.373862    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acab457-5f11-4d97-9467-1e86f8d6f99e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-sffmg\" (UID: \"0acab457-5f11-4d97-9467-1e86f8d6f99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"
Oct 30 20:15:00.379432 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.379410    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f1aa42-openshift-default-dockercfg-lp95x\""
Oct 30 20:15:00.431374 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.431292    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6857f\" (UniqueName: \"kubernetes.io/projected/0acab457-5f11-4d97-9467-1e86f8d6f99e-kube-api-access-6857f\") pod \"maas-api-key-cleanup-29889855-sffmg\" (UID: \"0acab457-5f11-4d97-9467-1e86f8d6f99e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"
Oct 30 20:15:00.474452 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.474421    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-socket\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.474643 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.474468    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-token\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.474643 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.474508    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istiod-ca-cert\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.474643 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.474549    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-data\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.474843 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.474736    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-envoy\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.474843 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.474805    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlg7\" (UniqueName: \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-kube-api-access-bzlg7\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.474966 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.474886    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-credential-socket\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.474966 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.474908    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-socket\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.474966 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.474914    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-certs\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.475139 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.474981    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-podinfo\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.475206 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.475172    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-certs\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.475269 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.475212    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-credential-socket\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.475413 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.475386    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istiod-ca-cert\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.475547 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.475419    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-data\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.477134 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.477109    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-envoy\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.477571 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.477550    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-podinfo\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.481519 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.481502    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"
Oct 30 20:15:00.532455 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.532424    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-token\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.539887 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.539807    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlg7\" (UniqueName: \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-kube-api-access-bzlg7\") pod \"e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") " pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.645370 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.645335    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:00.862665 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:00.862640    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"]
Oct 30 20:15:00.864688 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:15:00.864660    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acab457_5f11_4d97_9467_1e86f8d6f99e.slice/crio-056c52703c827931e37e83aa2fc16f0091be4057c088deb537a840de08ba2027 WatchSource:0}: Error finding container 056c52703c827931e37e83aa2fc16f0091be4057c088deb537a840de08ba2027: Status 404 returned error can't find the container with id 056c52703c827931e37e83aa2fc16f0091be4057c088deb537a840de08ba2027
Oct 30 20:15:01.021563 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:01.021540    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"]
Oct 30 20:15:01.023464 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:15:01.023439    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e9bef3_e55a_426a_81ca_328cb50a2f9a.slice/crio-c96ebe74aed6daa12cf256f6c94e478f842091b5752eaa97f67c2991e2999d1f WatchSource:0}: Error finding container c96ebe74aed6daa12cf256f6c94e478f842091b5752eaa97f67c2991e2999d1f: Status 404 returned error can't find the container with id c96ebe74aed6daa12cf256f6c94e478f842091b5752eaa97f67c2991e2999d1f
Oct 30 20:15:01.025687 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:01.025656    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:15:01.025782 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:01.025729    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:15:01.025782 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:01.025755    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:15:01.489179 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:01.489140    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg" event={"ID":"0acab457-5f11-4d97-9467-1e86f8d6f99e","Type":"ContainerStarted","Data":"105c99ff83173c526f33c2ab8228e5a00688fbaf45ce5fc5203406f18d82db31"}
Oct 30 20:15:01.489612 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:01.489220    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg" event={"ID":"0acab457-5f11-4d97-9467-1e86f8d6f99e","Type":"ContainerStarted","Data":"056c52703c827931e37e83aa2fc16f0091be4057c088deb537a840de08ba2027"}
Oct 30 20:15:01.490749 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:01.490724    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" event={"ID":"c5e9bef3-e55a-426a-81ca-328cb50a2f9a","Type":"ContainerStarted","Data":"fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b"}
Oct 30 20:15:01.490825 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:01.490769    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" event={"ID":"c5e9bef3-e55a-426a-81ca-328cb50a2f9a","Type":"ContainerStarted","Data":"c96ebe74aed6daa12cf256f6c94e478f842091b5752eaa97f67c2991e2999d1f"}
Oct 30 20:15:01.621278 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:01.621233    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg" podStartSLOduration=1.621218617 podStartE2EDuration="1.621218617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:01.619715053 +0000 UTC m=+3548.830932196" watchObservedRunningTime="2026-10-30 20:15:01.621218617 +0000 UTC m=+3548.832435763"
Oct 30 20:15:01.645931 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:01.645909    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:01.739923 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:01.739795    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" podStartSLOduration=1.739778275 podStartE2EDuration="1.739778275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:01.738310895 +0000 UTC m=+3548.949528040" watchObservedRunningTime="2026-10-30 20:15:01.739778275 +0000 UTC m=+3548.950995417"
Oct 30 20:15:02.497103 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:02.497069    4420 generic.go:356] "Generic (PLEG): container finished" podID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerID="105c99ff83173c526f33c2ab8228e5a00688fbaf45ce5fc5203406f18d82db31" exitCode=60
Oct 30 20:15:02.497545 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:02.497162    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg" event={"ID":"0acab457-5f11-4d97-9467-1e86f8d6f99e","Type":"ContainerDied","Data":"105c99ff83173c526f33c2ab8228e5a00688fbaf45ce5fc5203406f18d82db31"}
Oct 30 20:15:02.497545 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:02.497300    4420 scope.go:122] "RemoveContainer" containerID="105c99ff83173c526f33c2ab8228e5a00688fbaf45ce5fc5203406f18d82db31"
Oct 30 20:15:02.651371 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:02.651344    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:03.504367 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:03.504337    4420 generic.go:356] "Generic (PLEG): container finished" podID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerID="6b25b8b5bf63ce62ecb239a3c23ea22dc58327a3c51989ab35966cd9057236d0" exitCode=60
Oct 30 20:15:03.504796 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:03.504486    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg" event={"ID":"0acab457-5f11-4d97-9467-1e86f8d6f99e","Type":"ContainerDied","Data":"6b25b8b5bf63ce62ecb239a3c23ea22dc58327a3c51989ab35966cd9057236d0"}
Oct 30 20:15:03.504796 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:03.504546    4420 scope.go:122] "RemoveContainer" containerID="105c99ff83173c526f33c2ab8228e5a00688fbaf45ce5fc5203406f18d82db31"
Oct 30 20:15:03.504796 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:03.504635    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:03.504796 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:03.504762    4420 scope.go:122] "RemoveContainer" containerID="6b25b8b5bf63ce62ecb239a3c23ea22dc58327a3c51989ab35966cd9057236d0"
Oct 30 20:15:03.505072 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:15:03.505053    4420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-sffmg_odh-ai-gateway-infra(0acab457-5f11-4d97-9467-1e86f8d6f99e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e"
Oct 30 20:15:03.505901 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:03.505869    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:04.511569 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:04.511497    4420 scope.go:122] "RemoveContainer" containerID="6b25b8b5bf63ce62ecb239a3c23ea22dc58327a3c51989ab35966cd9057236d0"
Oct 30 20:15:04.511960 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:15:04.511702    4420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-sffmg_odh-ai-gateway-infra(0acab457-5f11-4d97-9467-1e86f8d6f99e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e"
Oct 30 20:15:05.197021 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:15:05.196982    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:15:10.296371 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.296336    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"]
Oct 30 20:15:10.301303 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.301286    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.356540 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.356520    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"]
Oct 30 20:15:10.366633 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.366610    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe7be38-0bc7-4d83-adea-478f4698ece5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.366743 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.366678    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.366872 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.366848    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.366925 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.366910    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb9vc\" (UniqueName: \"kubernetes.io/projected/0fe7be38-0bc7-4d83-adea-478f4698ece5-kube-api-access-cb9vc\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.403169 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.403147    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f1aa42-serving-cert\""
Oct 30 20:15:10.403265 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.403183    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:15:10.403265 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.403225    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8c9cv\""
Oct 30 20:15:10.467762 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.467735    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe7be38-0bc7-4d83-adea-478f4698ece5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.467998 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.467801    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.467998 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.467885    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.467998 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.467930    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb9vc\" (UniqueName: \"kubernetes.io/projected/0fe7be38-0bc7-4d83-adea-478f4698ece5-kube-api-access-cb9vc\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.467998 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:15:10.467962    4420 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-serving-cert: secret "maas-api-e2e-worker-single-main-f1aa42-serving-cert" not found
Oct 30 20:15:10.468211 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:15:10.468024    4420 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-maas-api-tls podName:0fe7be38-0bc7-4d83-adea-478f4698ece5 nodeName:}" failed. No retries permitted until 2026-10-30 20:15:10.968003582 +0000 UTC m=+3558.179220714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-maas-api-tls") pod "maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt" (UID: "0fe7be38-0bc7-4d83-adea-478f4698ece5") : secret "maas-api-e2e-worker-single-main-f1aa42-serving-cert" not found
Oct 30 20:15:10.468402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.468384    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe7be38-0bc7-4d83-adea-478f4698ece5-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.470283 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.470266    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.528074 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.528046    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb9vc\" (UniqueName: \"kubernetes.io/projected/0fe7be38-0bc7-4d83-adea-478f4698ece5-kube-api-access-cb9vc\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.972728 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.972696    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:10.975281 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:10.975260    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:11.213997 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:11.213962    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:11.595912 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:11.595885    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"]
Oct 30 20:15:11.597967 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:15:11.597931    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe7be38_0bc7_4d83_adea_478f4698ece5.slice/crio-182eda200e98392e0905a8deeabc2f9abe9cfab404a5fc8b1411d4db9d3a605d WatchSource:0}: Error finding container 182eda200e98392e0905a8deeabc2f9abe9cfab404a5fc8b1411d4db9d3a605d: Status 404 returned error can't find the container with id 182eda200e98392e0905a8deeabc2f9abe9cfab404a5fc8b1411d4db9d3a605d
Oct 30 20:15:11.599396 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:11.599378    4420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:15:12.552416 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:12.552380    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt" event={"ID":"0fe7be38-0bc7-4d83-adea-478f4698ece5","Type":"ContainerStarted","Data":"c59f42557baf79fef0124319bdff3ef47851a88ea6b1b6f993887171500d81fa"}
Oct 30 20:15:12.552542 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:12.552430    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt" event={"ID":"0fe7be38-0bc7-4d83-adea-478f4698ece5","Type":"ContainerStarted","Data":"182eda200e98392e0905a8deeabc2f9abe9cfab404a5fc8b1411d4db9d3a605d"}
Oct 30 20:15:12.552542 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:12.552496    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:12.669202 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:12.669162    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt" podStartSLOduration=1.885049465 podStartE2EDuration="2.669149887s" totalImagesPullingTime="784.100422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:12.666695546 +0000 UTC m=+3559.877912690" watchObservedRunningTime="2026-10-30 20:15:12.669149887 +0000 UTC m=+3559.880367035"
Oct 30 20:15:16.889357 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:16.889323    4420 scope.go:122] "RemoveContainer" containerID="6b25b8b5bf63ce62ecb239a3c23ea22dc58327a3c51989ab35966cd9057236d0"
Oct 30 20:15:17.593183 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:17.593101    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg" event={"ID":"0acab457-5f11-4d97-9467-1e86f8d6f99e","Type":"ContainerStarted","Data":"7788e901a971ac41658d8be45e08bbdd59c64d121f8ce85931e31e1c5d91c5ab"}
Oct 30 20:15:18.583044 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:18.583008    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:15:18.600333 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:18.600304    4420 generic.go:356] "Generic (PLEG): container finished" podID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerID="7788e901a971ac41658d8be45e08bbdd59c64d121f8ce85931e31e1c5d91c5ab" exitCode=60
Oct 30 20:15:18.600462 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:18.600375    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg" event={"ID":"0acab457-5f11-4d97-9467-1e86f8d6f99e","Type":"ContainerDied","Data":"7788e901a971ac41658d8be45e08bbdd59c64d121f8ce85931e31e1c5d91c5ab"}
Oct 30 20:15:18.600462 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:18.600428    4420 scope.go:122] "RemoveContainer" containerID="6b25b8b5bf63ce62ecb239a3c23ea22dc58327a3c51989ab35966cd9057236d0"
Oct 30 20:15:18.600685 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:18.600663    4420 scope.go:122] "RemoveContainer" containerID="7788e901a971ac41658d8be45e08bbdd59c64d121f8ce85931e31e1c5d91c5ab"
Oct 30 20:15:18.600933 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:15:18.600909    4420 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-sffmg_odh-ai-gateway-infra(0acab457-5f11-4d97-9467-1e86f8d6f99e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e"
Oct 30 20:15:18.735534 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:18.735501    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"]
Oct 30 20:15:19.796681 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:19.796659    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"
Oct 30 20:15:19.872392 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:19.872328    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0acab457-5f11-4d97-9467-1e86f8d6f99e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acab457-5f11-4d97-9467-1e86f8d6f99e-maas-api-service-ca\") pod \"0acab457-5f11-4d97-9467-1e86f8d6f99e\" (UID: \"0acab457-5f11-4d97-9467-1e86f8d6f99e\") "
Oct 30 20:15:19.876612 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:19.875976    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0acab457-5f11-4d97-9467-1e86f8d6f99e-kube-api-access-6857f\" (UniqueName: \"kubernetes.io/projected/0acab457-5f11-4d97-9467-1e86f8d6f99e-kube-api-access-6857f\") pod \"0acab457-5f11-4d97-9467-1e86f8d6f99e\" (UID: \"0acab457-5f11-4d97-9467-1e86f8d6f99e\") "
Oct 30 20:15:19.882984 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:19.882951    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0acab457-5f11-4d97-9467-1e86f8d6f99e-maas-api-service-ca" pod "0acab457-5f11-4d97-9467-1e86f8d6f99e" (UID: "0acab457-5f11-4d97-9467-1e86f8d6f99e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:15:19.884719 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:19.884695    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0acab457-5f11-4d97-9467-1e86f8d6f99e-kube-api-access-6857f" pod "0acab457-5f11-4d97-9467-1e86f8d6f99e" (UID: "0acab457-5f11-4d97-9467-1e86f8d6f99e"). InnerVolumeSpecName "kube-api-access-6857f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:15:19.980603 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:19.980567    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6857f\" (UniqueName: \"kubernetes.io/projected/0acab457-5f11-4d97-9467-1e86f8d6f99e-kube-api-access-6857f\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:19.980850 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:19.980808    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0acab457-5f11-4d97-9467-1e86f8d6f99e-maas-api-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:15:20.613913 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:20.613878    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg" event={"ID":"0acab457-5f11-4d97-9467-1e86f8d6f99e","Type":"ContainerDied","Data":"056c52703c827931e37e83aa2fc16f0091be4057c088deb537a840de08ba2027"}
Oct 30 20:15:20.614110 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:20.613939    4420 scope.go:122] "RemoveContainer" containerID="7788e901a971ac41658d8be45e08bbdd59c64d121f8ce85931e31e1c5d91c5ab"
Oct 30 20:15:20.614110 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:20.613895    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"
Oct 30 20:15:20.788711 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:20.788677    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"]
Oct 30 20:15:20.791140 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:20.791118    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-sffmg"]
Oct 30 20:15:20.895476 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:20.895448    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e" path="/var/lib/kubelet/pods/0acab457-5f11-4d97-9467-1e86f8d6f99e/volumes"
Oct 30 20:15:25.643175 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:25.643144    4420 generic.go:356] "Generic (PLEG): container finished" podID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerID="fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b" exitCode=0
Oct 30 20:15:25.643175 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:25.643177    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" event={"ID":"c5e9bef3-e55a-426a-81ca-328cb50a2f9a","Type":"ContainerDied","Data":"fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b"}
Oct 30 20:15:25.643607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:25.643481    4420 scope.go:122] "RemoveContainer" containerID="fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b"
Oct 30 20:15:25.644601 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:25.644572    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:15:25.644687 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:25.644636    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:15:25.644722 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:25.644685    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:15:26.650146 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:26.650115    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" event={"ID":"c5e9bef3-e55a-426a-81ca-328cb50a2f9a","Type":"ContainerStarted","Data":"180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6"}
Oct 30 20:15:27.645569 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:27.645536    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:27.645754 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:27.645724    4420 patch_prober.go:29] interesting pod/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:15:27.645872 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:27.645789    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 20:15:28.645979 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:28.645949    4420 patch_prober.go:29] interesting pod/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:15:28.646380 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:28.646000    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 20:15:29.646213 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:29.646146    4420 patch_prober.go:29] interesting pod/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:15:29.646629 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:29.646232    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 20:15:30.646139 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:30.646107    4420 patch_prober.go:29] interesting pod/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 30 20:15:30.646373 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:30.646171    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 30 20:15:31.650262 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:31.650237    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:31.650647 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:31.650486    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:31.651193 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:31.651174    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:15:35.195344 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:15:35.195297    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:15:39.048936 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:39.048903    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"]
Oct 30 20:15:39.098035 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:39.098010    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-g22qj"]
Oct 30 20:15:40.894309 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:40.894274    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bac95397-9748-41ff-b3bb-132d415ec32d" path="/var/lib/kubelet/pods/bac95397-9748-41ff-b3bb-132d415ec32d/volumes"
Oct 30 20:15:49.052206 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:49.052177    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"]
Oct 30 20:15:49.101436 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:49.101412    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4t7bf"]
Oct 30 20:15:50.894006 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:50.893972    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936c7002-06bb-4d0f-b7e0-9cae85e1f76c" path="/var/lib/kubelet/pods/936c7002-06bb-4d0f-b7e0-9cae85e1f76c/volumes"
Oct 30 20:15:53.908304 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:53.908277    4420 scope.go:122] "RemoveContainer" containerID="e71fd5a3a6341c974dbf6fcd27e4feb5ba12cf77ab1aa7def1f8e832cab2c03b"
Oct 30 20:15:53.916762 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:15:53.916743    4420 scope.go:122] "RemoveContainer" containerID="a59cb30b9b45aad997bd0de16c644ac9306ba69067abf9cd96eeb20194151fcf"
Oct 30 20:16:05.195228 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:16:05.195192    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:16:24.047442 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:24.047406    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"]
Oct 30 20:16:24.096318 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:24.096282    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n78mp"]
Oct 30 20:16:24.894738 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:24.894704    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d5f5c9-85a7-4b63-96b5-a769ff98cfd7" path="/var/lib/kubelet/pods/70d5f5c9-85a7-4b63-96b5-a769ff98cfd7/volumes"
Oct 30 20:16:35.196282 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:16:35.196202    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:16:52.968303 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:52.968264    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"]
Oct 30 20:16:52.968739 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:52.968456    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt" podUID="0fe7be38-0bc7-4d83-adea-478f4698ece5" containerName="maas-api" containerID="cri-o://c59f42557baf79fef0124319bdff3ef47851a88ea6b1b6f993887171500d81fa" gracePeriod=30
Oct 30 20:16:53.558485 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:53.558447    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt" podUID="0fe7be38-0bc7-4d83-adea-478f4698ece5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.137:8443/health\": dial tcp 10.133.0.137:8443: connect: connection refused"
Oct 30 20:16:53.943268 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:53.943238    4420 scope.go:122] "RemoveContainer" containerID="2a44b8a1225a80dfc0d3e738c9ca78b085657f06450275286f9688c71b2c619a"
Oct 30 20:16:53.953257 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:53.953234    4420 scope.go:122] "RemoveContainer" containerID="a027ce5b9e20ee6e17b6d083ab30820d1851fd4d5c42bd9c342463e7d591f01c"
Oct 30 20:16:54.116491 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.116455    4420 generic.go:356] "Generic (PLEG): container finished" podID="0fe7be38-0bc7-4d83-adea-478f4698ece5" containerID="c59f42557baf79fef0124319bdff3ef47851a88ea6b1b6f993887171500d81fa" exitCode=0
Oct 30 20:16:54.116878 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.116518    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt" event={"ID":"0fe7be38-0bc7-4d83-adea-478f4698ece5","Type":"ContainerDied","Data":"c59f42557baf79fef0124319bdff3ef47851a88ea6b1b6f993887171500d81fa"}
Oct 30 20:16:54.219532 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.219509    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:16:54.267026 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.267003    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe7be38-0bc7-4d83-adea-478f4698ece5-kube-api-access-cb9vc\" (UniqueName: \"kubernetes.io/projected/0fe7be38-0bc7-4d83-adea-478f4698ece5-kube-api-access-cb9vc\") pod \"0fe7be38-0bc7-4d83-adea-478f4698ece5\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") "
Oct 30 20:16:54.267155 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.267041    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-maas-api-tls\") pod \"0fe7be38-0bc7-4d83-adea-478f4698ece5\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") "
Oct 30 20:16:54.267155 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.267090    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe7be38-0bc7-4d83-adea-478f4698ece5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe7be38-0bc7-4d83-adea-478f4698ece5-openshift-service-ca\") pod \"0fe7be38-0bc7-4d83-adea-478f4698ece5\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") "
Oct 30 20:16:54.267155 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.267150    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-metrics-certs\") pod \"0fe7be38-0bc7-4d83-adea-478f4698ece5\" (UID: \"0fe7be38-0bc7-4d83-adea-478f4698ece5\") "
Oct 30 20:16:54.267496 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.267464    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe7be38-0bc7-4d83-adea-478f4698ece5-openshift-service-ca" pod "0fe7be38-0bc7-4d83-adea-478f4698ece5" (UID: "0fe7be38-0bc7-4d83-adea-478f4698ece5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:16:54.269319 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.269292    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-metrics-certs" pod "0fe7be38-0bc7-4d83-adea-478f4698ece5" (UID: "0fe7be38-0bc7-4d83-adea-478f4698ece5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:54.269412 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.269356    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-maas-api-tls" pod "0fe7be38-0bc7-4d83-adea-478f4698ece5" (UID: "0fe7be38-0bc7-4d83-adea-478f4698ece5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:16:54.269412 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.269383    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe7be38-0bc7-4d83-adea-478f4698ece5-kube-api-access-cb9vc" pod "0fe7be38-0bc7-4d83-adea-478f4698ece5" (UID: "0fe7be38-0bc7-4d83-adea-478f4698ece5"). InnerVolumeSpecName "kube-api-access-cb9vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:16:54.368125 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.368093    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb9vc\" (UniqueName: \"kubernetes.io/projected/0fe7be38-0bc7-4d83-adea-478f4698ece5-kube-api-access-cb9vc\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:54.368125 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.368124    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:54.368331 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.368137    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe7be38-0bc7-4d83-adea-478f4698ece5-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:54.368331 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:54.368149    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fe7be38-0bc7-4d83-adea-478f4698ece5-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:16:55.122906 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:55.122831    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"
Oct 30 20:16:55.122906 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:55.122854    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt" event={"ID":"0fe7be38-0bc7-4d83-adea-478f4698ece5","Type":"ContainerDied","Data":"182eda200e98392e0905a8deeabc2f9abe9cfab404a5fc8b1411d4db9d3a605d"}
Oct 30 20:16:55.123462 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:55.122932    4420 scope.go:122] "RemoveContainer" containerID="c59f42557baf79fef0124319bdff3ef47851a88ea6b1b6f993887171500d81fa"
Oct 30 20:16:55.288457 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:55.288424    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"]
Oct 30 20:16:55.337576 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:55.337543    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f1aa42-fdd9b658d-n6tlt"]
Oct 30 20:16:56.894026 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:56.893997    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe7be38-0bc7-4d83-adea-478f4698ece5" path="/var/lib/kubelet/pods/0fe7be38-0bc7-4d83-adea-478f4698ece5/volumes"
Oct 30 20:16:59.048903 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:59.048875    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"]
Oct 30 20:16:59.097765 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:16:59.097733    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-2xbxn"]
Oct 30 20:17:00.894595 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:00.894560    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5893827-a4b0-41ef-9eab-e1dc37cfbe2a" path="/var/lib/kubelet/pods/e5893827-a4b0-41ef-9eab-e1dc37cfbe2a/volumes"
Oct 30 20:17:02.278985 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:02.278952    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"]
Oct 30 20:17:02.279372 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:02.279186    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerName="istio-proxy" containerID="cri-o://180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6" gracePeriod=30
Oct 30 20:17:03.892648 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:03.892618    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"]
Oct 30 20:17:03.941469 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:03.941434    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-n46vj"]
Oct 30 20:17:04.013711 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.013681    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"]
Oct 30 20:17:04.014155 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.014136    4420 cpu_manager.go:414] "containerMap: removing container" podUID="0fe7be38-0bc7-4d83-adea-478f4698ece5" containerName="maas-api"
Oct 30 20:17:04.014210 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.014158    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe7be38-0bc7-4d83-adea-478f4698ece5" containerName="maas-api"
Oct 30 20:17:04.014210 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.014174    4420 cpu_manager.go:414] "containerMap: removing container" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerName="cleanup"
Oct 30 20:17:04.014210 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.014179    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerName="cleanup"
Oct 30 20:17:04.014210 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.014194    4420 cpu_manager.go:414] "containerMap: removing container" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerName="cleanup"
Oct 30 20:17:04.014210 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.014199    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerName="cleanup"
Oct 30 20:17:04.014400 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.014268    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe7be38-0bc7-4d83-adea-478f4698ece5" containerName="maas-api"
Oct 30 20:17:04.014400 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.014276    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerName="cleanup"
Oct 30 20:17:04.014400 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.014284    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerName="cleanup"
Oct 30 20:17:04.024235 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.023094    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:04.052744 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.052714    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xw5rq\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:04.052907 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.052882    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xw5rq\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:04.052953 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.052912    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvcwr\" (UniqueName: \"kubernetes.io/projected/2043e99f-17e4-4e47-9c74-e5dd7c390277-kube-api-access-lvcwr\") pod \"maas-api-revoke-sub-3d4156f1-xw5rq\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:04.074589 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.074563    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"]
Oct 30 20:17:04.075199 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.075176    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:17:04.075324 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.075248    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:17:04.153765 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.153688    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvcwr\" (UniqueName: \"kubernetes.io/projected/2043e99f-17e4-4e47-9c74-e5dd7c390277-kube-api-access-lvcwr\") pod \"maas-api-revoke-sub-3d4156f1-xw5rq\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:04.153922 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.153773    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xw5rq\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:04.153970 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.153940    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xw5rq\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:04.154646 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.154617    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xw5rq\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:04.156373 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.156355    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xw5rq\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:04.220181 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.220147    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvcwr\" (UniqueName: \"kubernetes.io/projected/2043e99f-17e4-4e47-9c74-e5dd7c390277-kube-api-access-lvcwr\") pod \"maas-api-revoke-sub-3d4156f1-xw5rq\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:04.337080 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.337053    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:04.712996 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.712909    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"]
Oct 30 20:17:04.715560 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:17:04.715531    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2043e99f_17e4_4e47_9c74_e5dd7c390277.slice/crio-cb989260faf21b4218366a84954f2cee1cc7be12b87604ff5064f0f341013283 WatchSource:0}: Error finding container cb989260faf21b4218366a84954f2cee1cc7be12b87604ff5064f0f341013283: Status 404 returned error can't find the container with id cb989260faf21b4218366a84954f2cee1cc7be12b87604ff5064f0f341013283
Oct 30 20:17:04.894951 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:04.894921    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9" path="/var/lib/kubelet/pods/2c4c6289-6c5a-45d4-a7ed-ce8338b8c5a9/volumes"
Oct 30 20:17:05.175609 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:05.175581    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq" event={"ID":"2043e99f-17e4-4e47-9c74-e5dd7c390277","Type":"ContainerStarted","Data":"3690e501e888c4be8f131dc1649beb7fb9b280a109c646f386896bcf3f94542a"}
Oct 30 20:17:05.175746 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:05.175634    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq" event={"ID":"2043e99f-17e4-4e47-9c74-e5dd7c390277","Type":"ContainerStarted","Data":"cb989260faf21b4218366a84954f2cee1cc7be12b87604ff5064f0f341013283"}
Oct 30 20:17:05.196041 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:17:05.196011    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:17:05.288994 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:05.288726    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq" podStartSLOduration=2.288712229 podStartE2EDuration="2.288712229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:05.288044658 +0000 UTC m=+3672.499261816" watchObservedRunningTime="2026-10-30 20:17:05.288712229 +0000 UTC m=+3672.499929373"
Oct 30 20:17:06.181739 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:06.181706    4420 generic.go:356] "Generic (PLEG): container finished" podID="2043e99f-17e4-4e47-9c74-e5dd7c390277" containerID="3690e501e888c4be8f131dc1649beb7fb9b280a109c646f386896bcf3f94542a" exitCode=0
Oct 30 20:17:06.182144 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:06.181789    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq" event={"ID":"2043e99f-17e4-4e47-9c74-e5dd7c390277","Type":"ContainerDied","Data":"3690e501e888c4be8f131dc1649beb7fb9b280a109c646f386896bcf3f94542a"}
Oct 30 20:17:07.407035 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.407009    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:07.488740 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.488711    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2043e99f-17e4-4e47-9c74-e5dd7c390277-kube-api-access-lvcwr\" (UniqueName: \"kubernetes.io/projected/2043e99f-17e4-4e47-9c74-e5dd7c390277-kube-api-access-lvcwr\") pod \"2043e99f-17e4-4e47-9c74-e5dd7c390277\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") "
Oct 30 20:17:07.488951 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.488773    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-openshift-ca\") pod \"2043e99f-17e4-4e47-9c74-e5dd7c390277\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") "
Oct 30 20:17:07.489006 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.488965    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-xks-ca\") pod \"2043e99f-17e4-4e47-9c74-e5dd7c390277\" (UID: \"2043e99f-17e4-4e47-9c74-e5dd7c390277\") "
Oct 30 20:17:07.489210 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.489183    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-openshift-ca" pod "2043e99f-17e4-4e47-9c74-e5dd7c390277" (UID: "2043e99f-17e4-4e47-9c74-e5dd7c390277"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:07.489344 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.489324    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.492229 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.492199    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-xks-ca" pod "2043e99f-17e4-4e47-9c74-e5dd7c390277" (UID: "2043e99f-17e4-4e47-9c74-e5dd7c390277"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:17:07.492335 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.492268    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2043e99f-17e4-4e47-9c74-e5dd7c390277-kube-api-access-lvcwr" pod "2043e99f-17e4-4e47-9c74-e5dd7c390277" (UID: "2043e99f-17e4-4e47-9c74-e5dd7c390277"). InnerVolumeSpecName "kube-api-access-lvcwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:07.526915 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.526890    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:17:07.590617 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.590589    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istiod-ca-cert\") pod \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") "
Oct 30 20:17:07.590769 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.590636    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-data\") pod \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") "
Oct 30 20:17:07.590769 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.590663    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-credential-socket\") pod \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") "
Oct 30 20:17:07.590769 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.590699    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-envoy\") pod \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") "
Oct 30 20:17:07.590769 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.590740    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-podinfo\") pod \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") "
Oct 30 20:17:07.590769 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.590766    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-socket\") pod \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") "
Oct 30 20:17:07.591042 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.590800    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-certs\") pod \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") "
Oct 30 20:17:07.591042 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.590893    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-token\" (UniqueName: \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-token\") pod \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") "
Oct 30 20:17:07.591042 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.590948    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-kube-api-access-bzlg7\" (UniqueName: \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-kube-api-access-bzlg7\") pod \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\" (UID: \"c5e9bef3-e55a-426a-81ca-328cb50a2f9a\") "
Oct 30 20:17:07.591042 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.591003    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-data" pod "c5e9bef3-e55a-426a-81ca-328cb50a2f9a" (UID: "c5e9bef3-e55a-426a-81ca-328cb50a2f9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:07.591248 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.591054    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istiod-ca-cert" pod "c5e9bef3-e55a-426a-81ca-328cb50a2f9a" (UID: "c5e9bef3-e55a-426a-81ca-328cb50a2f9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:17:07.591248 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.591199    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-credential-socket" pod "c5e9bef3-e55a-426a-81ca-328cb50a2f9a" (UID: "c5e9bef3-e55a-426a-81ca-328cb50a2f9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:07.591248 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.591210    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-socket" pod "c5e9bef3-e55a-426a-81ca-328cb50a2f9a" (UID: "c5e9bef3-e55a-426a-81ca-328cb50a2f9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:07.591402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.591274    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-certs" pod "c5e9bef3-e55a-426a-81ca-328cb50a2f9a" (UID: "c5e9bef3-e55a-426a-81ca-328cb50a2f9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:07.591402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.591350    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.591402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.591365    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2043e99f-17e4-4e47-9c74-e5dd7c390277-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.591402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.591375    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.591402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.591384    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.591402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.591392    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.591402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.591402    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvcwr\" (UniqueName: \"kubernetes.io/projected/2043e99f-17e4-4e47-9c74-e5dd7c390277-kube-api-access-lvcwr\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.593259 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.593234    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-podinfo" pod "c5e9bef3-e55a-426a-81ca-328cb50a2f9a" (UID: "c5e9bef3-e55a-426a-81ca-328cb50a2f9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:17:07.593358 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.593278    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-kube-api-access-bzlg7" pod "c5e9bef3-e55a-426a-81ca-328cb50a2f9a" (UID: "c5e9bef3-e55a-426a-81ca-328cb50a2f9a"). InnerVolumeSpecName "kube-api-access-bzlg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:07.593572 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.593556    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-token" pod "c5e9bef3-e55a-426a-81ca-328cb50a2f9a" (UID: "c5e9bef3-e55a-426a-81ca-328cb50a2f9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:17:07.593616 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.593569    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-envoy" pod "c5e9bef3-e55a-426a-81ca-328cb50a2f9a" (UID: "c5e9bef3-e55a-426a-81ca-328cb50a2f9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:17:07.692251 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.692218    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.692251 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.692243    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.692251 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.692254    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.692483 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.692263    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:07.692483 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:07.692274    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlg7\" (UniqueName: \"kubernetes.io/projected/c5e9bef3-e55a-426a-81ca-328cb50a2f9a-kube-api-access-bzlg7\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:17:08.193115 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.193085    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"
Oct 30 20:17:08.193315 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.193117    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq" event={"ID":"2043e99f-17e4-4e47-9c74-e5dd7c390277","Type":"ContainerDied","Data":"cb989260faf21b4218366a84954f2cee1cc7be12b87604ff5064f0f341013283"}
Oct 30 20:17:08.193315 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.193161    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb989260faf21b4218366a84954f2cee1cc7be12b87604ff5064f0f341013283"
Oct 30 20:17:08.195085 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.195062    4420 generic.go:356] "Generic (PLEG): container finished" podID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerID="180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6" exitCode=0
Oct 30 20:17:08.195206 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.195112    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" event={"ID":"c5e9bef3-e55a-426a-81ca-328cb50a2f9a","Type":"ContainerDied","Data":"180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6"}
Oct 30 20:17:08.195206 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.195129    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"
Oct 30 20:17:08.195206 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.195147    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b" event={"ID":"c5e9bef3-e55a-426a-81ca-328cb50a2f9a","Type":"ContainerDied","Data":"c96ebe74aed6daa12cf256f6c94e478f842091b5752eaa97f67c2991e2999d1f"}
Oct 30 20:17:08.195206 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.195175    4420 scope.go:122] "RemoveContainer" containerID="180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6"
Oct 30 20:17:08.205241 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.205220    4420 scope.go:122] "RemoveContainer" containerID="fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b"
Oct 30 20:17:08.214710 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.214687    4420 scope.go:122] "RemoveContainer" containerID="180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6"
Oct 30 20:17:08.215013 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:17:08.214995    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6\": container with ID starting with 180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6 not found: ID does not exist" containerID="180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6"
Oct 30 20:17:08.215084 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.215024    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6"} err="failed to get container status \"180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6\": rpc error: code = NotFound desc = could not find container \"180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6\": container with ID starting with 180a717634a5581d0c1dac784e9b183a8c65e6cc192e0daf55f02efa12c392d6 not found: ID does not exist"
Oct 30 20:17:08.215084 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.215040    4420 scope.go:122] "RemoveContainer" containerID="fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b"
Oct 30 20:17:08.215243 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:17:08.215228    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b\": container with ID starting with fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b not found: ID does not exist" containerID="fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b"
Oct 30 20:17:08.215284 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.215250    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b"} err="failed to get container status \"fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b\": rpc error: code = NotFound desc = could not find container \"fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b\": container with ID starting with fec7f0f006304f16e863c476cb3da5d4928e8db7892079acfdf86d96474d934b not found: ID does not exist"
Oct 30 20:17:08.357422 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.357398    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"]
Oct 30 20:17:08.374283 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.374254    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f1aa42-openshift-default-84f5c74dd5xd67b"]
Oct 30 20:17:08.894928 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:08.894897    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" path="/var/lib/kubelet/pods/c5e9bef3-e55a-426a-81ca-328cb50a2f9a/volumes"
Oct 30 20:17:35.196069 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:17:35.196033    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:17:43.058280 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:43.058247    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"]
Oct 30 20:17:43.107238 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:43.107203    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xdvbz"]
Oct 30 20:17:44.894633 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:44.894597    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf71c7d-5a29-498e-bf2e-0351c92092c8" path="/var/lib/kubelet/pods/fbf71c7d-5a29-498e-bf2e-0351c92092c8/volumes"
Oct 30 20:17:53.978493 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:17:53.978461    4420 scope.go:122] "RemoveContainer" containerID="a28552d75b5c966f898b727cf032d248c16f116f7bf517607c9c007b206f87f0"
Oct 30 20:18:05.196369 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:18:05.196293    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:18:08.047070 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:18:08.047039    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"]
Oct 30 20:18:08.095871 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:18:08.095844    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-stvpc"]
Oct 30 20:18:08.899335 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:18:08.899297    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1" path="/var/lib/kubelet/pods/2efd55bf-0c06-4e3f-8d5f-7c9ad92d3bd1/volumes"
Oct 30 20:18:35.196838 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:18:35.196775    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:18:54.007280 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:18:54.007248    4420 scope.go:122] "RemoveContainer" containerID="5897912d0b39d5ff2d5a139de30f7f0a35025af91e95613369743c4e5c16c919"
Oct 30 20:18:54.015671 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:18:54.015650    4420 scope.go:122] "RemoveContainer" containerID="456dd240d2e76d5301303ce1b2c6b45b8a5ac49cf028bddc4ef4036a6c7254bd"
Oct 30 20:19:05.195333 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:19:05.195295    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:19:18.368697 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.368664    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"]
Oct 30 20:19:18.369165 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.369150    4420 cpu_manager.go:414] "containerMap: removing container" podUID="2043e99f-17e4-4e47-9c74-e5dd7c390277" containerName="revoke-keys"
Oct 30 20:19:18.369209 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.369167    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2043e99f-17e4-4e47-9c74-e5dd7c390277" containerName="revoke-keys"
Oct 30 20:19:18.369209 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.369177    4420 cpu_manager.go:414] "containerMap: removing container" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerName="istio-proxy"
Oct 30 20:19:18.369209 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.369182    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerName="istio-proxy"
Oct 30 20:19:18.369209 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.369199    4420 cpu_manager.go:414] "containerMap: removing container" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerName="istio-proxy"
Oct 30 20:19:18.369209 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.369204    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerName="istio-proxy"
Oct 30 20:19:18.369363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.369220    4420 cpu_manager.go:414] "containerMap: removing container" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerName="cleanup"
Oct 30 20:19:18.369363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.369224    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerName="cleanup"
Oct 30 20:19:18.369363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.369295    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerName="istio-proxy"
Oct 30 20:19:18.369363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.369306    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="0acab457-5f11-4d97-9467-1e86f8d6f99e" containerName="cleanup"
Oct 30 20:19:18.369363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.369316    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="2043e99f-17e4-4e47-9c74-e5dd7c390277" containerName="revoke-keys"
Oct 30 20:19:18.372467 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.372452    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:18.429607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.429584    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"]
Oct 30 20:19:18.451650 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.451626    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gpcht\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:18.451786 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.451679    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gpcht\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:18.451885 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.451851    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp7h6\" (UniqueName: \"kubernetes.io/projected/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-kube-api-access-jp7h6\") pod \"maas-api-revoke-sub-d576e68b-gpcht\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:18.475792 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.475766    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:18.476097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.476071    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:19:18.553030 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.553007    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gpcht\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:18.553135 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.553067    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp7h6\" (UniqueName: \"kubernetes.io/projected/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-kube-api-access-jp7h6\") pod \"maas-api-revoke-sub-d576e68b-gpcht\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:18.553135 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.553093    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gpcht\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:18.553596 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.553575    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-gpcht\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:18.555450 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.555432    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-gpcht\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:18.609656 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.609630    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp7h6\" (UniqueName: \"kubernetes.io/projected/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-kube-api-access-jp7h6\") pod \"maas-api-revoke-sub-d576e68b-gpcht\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:18.684688 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:18.684664    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:19.055086 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:19.055014    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"]
Oct 30 20:19:19.058124 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:19:19.058094    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a3f6ef_b26d_4e8e_a142_b9ba15bad540.slice/crio-214b0ac5de0297b7352c253cf42d6f743e0bc1627835b36191cff943dc12f610 WatchSource:0}: Error finding container 214b0ac5de0297b7352c253cf42d6f743e0bc1627835b36191cff943dc12f610: Status 404 returned error can't find the container with id 214b0ac5de0297b7352c253cf42d6f743e0bc1627835b36191cff943dc12f610
Oct 30 20:19:19.864067 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:19.864031    4420 generic.go:356] "Generic (PLEG): container finished" podID="66a3f6ef-b26d-4e8e-a142-b9ba15bad540" containerID="0e10373375f17e1084213c5a59b82ca9fba41b905c027ca3173e8569ff9512f1" exitCode=0
Oct 30 20:19:19.864410 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:19.864118    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht" event={"ID":"66a3f6ef-b26d-4e8e-a142-b9ba15bad540","Type":"ContainerDied","Data":"0e10373375f17e1084213c5a59b82ca9fba41b905c027ca3173e8569ff9512f1"}
Oct 30 20:19:19.864410 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:19.864167    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht" event={"ID":"66a3f6ef-b26d-4e8e-a142-b9ba15bad540","Type":"ContainerStarted","Data":"214b0ac5de0297b7352c253cf42d6f743e0bc1627835b36191cff943dc12f610"}
Oct 30 20:19:21.004125 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.004101    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:21.074500 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.074480    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-kube-api-access-jp7h6\" (UniqueName: \"kubernetes.io/projected/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-kube-api-access-jp7h6\") pod \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") "
Oct 30 20:19:21.074609 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.074586    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-xks-ca\") pod \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") "
Oct 30 20:19:21.074609 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.074607    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-openshift-ca\") pod \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\" (UID: \"66a3f6ef-b26d-4e8e-a142-b9ba15bad540\") "
Oct 30 20:19:21.075013 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.074991    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-openshift-ca" pod "66a3f6ef-b26d-4e8e-a142-b9ba15bad540" (UID: "66a3f6ef-b26d-4e8e-a142-b9ba15bad540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:19:21.076598 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.076576    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-kube-api-access-jp7h6" pod "66a3f6ef-b26d-4e8e-a142-b9ba15bad540" (UID: "66a3f6ef-b26d-4e8e-a142-b9ba15bad540"). InnerVolumeSpecName "kube-api-access-jp7h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:19:21.076696 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.076632    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-xks-ca" pod "66a3f6ef-b26d-4e8e-a142-b9ba15bad540" (UID: "66a3f6ef-b26d-4e8e-a142-b9ba15bad540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:19:21.175400 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.175373    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp7h6\" (UniqueName: \"kubernetes.io/projected/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-kube-api-access-jp7h6\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:21.175400 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.175398    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:21.175560 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.175409    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a3f6ef-b26d-4e8e-a142-b9ba15bad540-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:19:21.875007 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.874972    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht" event={"ID":"66a3f6ef-b26d-4e8e-a142-b9ba15bad540","Type":"ContainerDied","Data":"214b0ac5de0297b7352c253cf42d6f743e0bc1627835b36191cff943dc12f610"}
Oct 30 20:19:21.875169 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.875018    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"
Oct 30 20:19:21.875169 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:21.875031    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="214b0ac5de0297b7352c253cf42d6f743e0bc1627835b36191cff943dc12f610"
Oct 30 20:19:35.195157 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:19:35.195074    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:19:54.038351 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:54.038321    4420 scope.go:122] "RemoveContainer" containerID="073934bd40fc2e16be02b24c15f2b05192af930151754536c46c7a55cc2bdd9f"
Oct 30 20:19:54.046340 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:54.046322    4420 scope.go:122] "RemoveContainer" containerID="4bf58335dbf8ba8a7ff37fe6f4de5c98eb3eebb40c0a8a3337df47cfa89e733b"
Oct 30 20:19:54.053753 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:54.053737    4420 scope.go:122] "RemoveContainer" containerID="0ed02c317a34a2b0bbf80179d0191fc748abef5c511f4d231f73dafaedf0b8cc"
Oct 30 20:19:57.651228 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.651187    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"]
Oct 30 20:19:57.651710 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.651615    4420 cpu_manager.go:414] "containerMap: removing container" podUID="66a3f6ef-b26d-4e8e-a142-b9ba15bad540" containerName="revoke-keys"
Oct 30 20:19:57.651710 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.651625    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a3f6ef-b26d-4e8e-a142-b9ba15bad540" containerName="revoke-keys"
Oct 30 20:19:57.651710 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.651688    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e9bef3-e55a-426a-81ca-328cb50a2f9a" containerName="istio-proxy"
Oct 30 20:19:57.651710 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.651700    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a3f6ef-b26d-4e8e-a142-b9ba15bad540" containerName="revoke-keys"
Oct 30 20:19:57.656119 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.656098    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:19:57.709643 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.709600    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:19:57.727129 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.727100    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r5ct\" (UniqueName: \"kubernetes.io/projected/7e58cd73-c383-435d-a14b-0dc45b19afad-kube-api-access-9r5ct\") pod \"maas-api-revoke-sub-d2bc93bc-88pbv\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:19:57.727274 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.727142    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-88pbv\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:19:57.727319 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.727273    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-88pbv\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:19:57.735020 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.734999    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:19:57.736789 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.736770    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"]
Oct 30 20:19:57.827972 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.827932    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r5ct\" (UniqueName: \"kubernetes.io/projected/7e58cd73-c383-435d-a14b-0dc45b19afad-kube-api-access-9r5ct\") pod \"maas-api-revoke-sub-d2bc93bc-88pbv\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:19:57.828073 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.827979    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-88pbv\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:19:57.828073 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.828028    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-88pbv\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:19:57.828636 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.828612    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-88pbv\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:19:57.830408 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.830392    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-88pbv\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:19:57.884908 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.884887    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r5ct\" (UniqueName: \"kubernetes.io/projected/7e58cd73-c383-435d-a14b-0dc45b19afad-kube-api-access-9r5ct\") pod \"maas-api-revoke-sub-d2bc93bc-88pbv\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:19:57.976089 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:57.976038    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:19:58.347181 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:58.347048    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"]
Oct 30 20:19:58.350107 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:19:58.350079    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e58cd73_c383_435d_a14b_0dc45b19afad.slice/crio-fd98b8217341d8a9b85555280f95a6bc330f6afe47ec1514ce6dbd9aa2516e75 WatchSource:0}: Error finding container fd98b8217341d8a9b85555280f95a6bc330f6afe47ec1514ce6dbd9aa2516e75: Status 404 returned error can't find the container with id fd98b8217341d8a9b85555280f95a6bc330f6afe47ec1514ce6dbd9aa2516e75
Oct 30 20:19:59.066266 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:59.066199    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv" event={"ID":"7e58cd73-c383-435d-a14b-0dc45b19afad","Type":"ContainerStarted","Data":"87bac46805cc6bc076f08d7c96dbc5423e74a82b33427cc213d864bcfc59d16b"}
Oct 30 20:19:59.066266 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:59.066254    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv" event={"ID":"7e58cd73-c383-435d-a14b-0dc45b19afad","Type":"ContainerStarted","Data":"fd98b8217341d8a9b85555280f95a6bc330f6afe47ec1514ce6dbd9aa2516e75"}
Oct 30 20:19:59.181256 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:19:59.181218    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv" podStartSLOduration=2.181204588 podStartE2EDuration="2.181204588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:19:59.178582951 +0000 UTC m=+3846.389800099" watchObservedRunningTime="2026-10-30 20:19:59.181204588 +0000 UTC m=+3846.392421732"
Oct 30 20:20:00.076199 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:00.076165    4420 generic.go:356] "Generic (PLEG): container finished" podID="7e58cd73-c383-435d-a14b-0dc45b19afad" containerID="87bac46805cc6bc076f08d7c96dbc5423e74a82b33427cc213d864bcfc59d16b" exitCode=0
Oct 30 20:20:00.076568 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:00.076246    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv" event={"ID":"7e58cd73-c383-435d-a14b-0dc45b19afad","Type":"ContainerDied","Data":"87bac46805cc6bc076f08d7c96dbc5423e74a82b33427cc213d864bcfc59d16b"}
Oct 30 20:20:01.207950 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:01.207926    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:20:01.259809 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:01.259784    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-xks-ca\") pod \"7e58cd73-c383-435d-a14b-0dc45b19afad\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") "
Oct 30 20:20:01.259963 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:01.259929    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-openshift-ca\") pod \"7e58cd73-c383-435d-a14b-0dc45b19afad\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") "
Oct 30 20:20:01.260003 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:01.259980    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e58cd73-c383-435d-a14b-0dc45b19afad-kube-api-access-9r5ct\" (UniqueName: \"kubernetes.io/projected/7e58cd73-c383-435d-a14b-0dc45b19afad-kube-api-access-9r5ct\") pod \"7e58cd73-c383-435d-a14b-0dc45b19afad\" (UID: \"7e58cd73-c383-435d-a14b-0dc45b19afad\") "
Oct 30 20:20:01.260315 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:01.260287    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-openshift-ca" pod "7e58cd73-c383-435d-a14b-0dc45b19afad" (UID: "7e58cd73-c383-435d-a14b-0dc45b19afad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:20:01.262147 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:01.262119    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-xks-ca" pod "7e58cd73-c383-435d-a14b-0dc45b19afad" (UID: "7e58cd73-c383-435d-a14b-0dc45b19afad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:20:01.262245 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:01.262200    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e58cd73-c383-435d-a14b-0dc45b19afad-kube-api-access-9r5ct" pod "7e58cd73-c383-435d-a14b-0dc45b19afad" (UID: "7e58cd73-c383-435d-a14b-0dc45b19afad"). InnerVolumeSpecName "kube-api-access-9r5ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:20:01.361673 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:01.361616    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:01.361673 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:01.361639    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r5ct\" (UniqueName: \"kubernetes.io/projected/7e58cd73-c383-435d-a14b-0dc45b19afad-kube-api-access-9r5ct\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:01.361673 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:01.361649    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e58cd73-c383-435d-a14b-0dc45b19afad-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:20:02.086384 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:02.086353    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv" event={"ID":"7e58cd73-c383-435d-a14b-0dc45b19afad","Type":"ContainerDied","Data":"fd98b8217341d8a9b85555280f95a6bc330f6afe47ec1514ce6dbd9aa2516e75"}
Oct 30 20:20:02.086384 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:02.086371    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"
Oct 30 20:20:02.086588 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:02.086396    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd98b8217341d8a9b85555280f95a6bc330f6afe47ec1514ce6dbd9aa2516e75"
Oct 30 20:20:05.195906 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:20:05.195865    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:20:10.309165 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.309137    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"]
Oct 30 20:20:10.309559 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.309546    4420 cpu_manager.go:414] "containerMap: removing container" podUID="7e58cd73-c383-435d-a14b-0dc45b19afad" containerName="revoke-keys"
Oct 30 20:20:10.309602 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.309560    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e58cd73-c383-435d-a14b-0dc45b19afad" containerName="revoke-keys"
Oct 30 20:20:10.309719 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.309692    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e58cd73-c383-435d-a14b-0dc45b19afad" containerName="revoke-keys"
Oct 30 20:20:10.313091 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.313074    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.368847 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.364852    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-14ab4d-openshift-default-dockercfg-5jvzr\""
Oct 30 20:20:10.416806 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.416776    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"]
Oct 30 20:20:10.438333 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.438302    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-envoy\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.438465 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.438351    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-credential-socket\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.438465 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.438422    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfx6j\" (UniqueName: \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-kube-api-access-mfx6j\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.438563 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.438549    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-certs\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.438604 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.438571    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27330f56-6fc8-4ff7-b32c-424e5330af41-istiod-ca-cert\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.438604 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.438590    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-token\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.438664 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.438613    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-podinfo\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.438713 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.438695    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-data\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.438748 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.438724    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-socket\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.539592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.539560    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-credential-socket\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.539592 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.539598    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfx6j\" (UniqueName: \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-kube-api-access-mfx6j\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.539949 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.539678    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-certs\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.539949 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.539716    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27330f56-6fc8-4ff7-b32c-424e5330af41-istiod-ca-cert\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.539949 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.539742    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-token\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.539949 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.539768    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-podinfo\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.539949 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.539875    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-data\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.539949 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.539912    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-socket\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.540265 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.539966    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-envoy\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.540265 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.540226    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-credential-socket\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.540374 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.540260    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-data\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.540442 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.540422    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-socket\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.540507 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.540483    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-certs\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.540675 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.540653    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27330f56-6fc8-4ff7-b32c-424e5330af41-istiod-ca-cert\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.542460 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.542431    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-envoy\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.542637 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.542621    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-podinfo\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.596171 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.596104    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-token\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.596292 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.596223    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfx6j\" (UniqueName: \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-kube-api-access-mfx6j\") pod \"e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") " pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:10.626629 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:10.626595    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:11.069732 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:11.069708    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"]
Oct 30 20:20:11.071745 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:20:11.071726    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27330f56_6fc8_4ff7_b32c_424e5330af41.slice/crio-c3af3387de315c5c66083683af2f4783c7817f857cb668f4dc3e4e14832878b9 WatchSource:0}: Error finding container c3af3387de315c5c66083683af2f4783c7817f857cb668f4dc3e4e14832878b9: Status 404 returned error can't find the container with id c3af3387de315c5c66083683af2f4783c7817f857cb668f4dc3e4e14832878b9
Oct 30 20:20:11.073843 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:11.073788    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:20:11.073939 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:11.073884    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:20:11.073939 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:11.073911    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:20:11.128448 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:11.128423    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" event={"ID":"27330f56-6fc8-4ff7-b32c-424e5330af41","Type":"ContainerStarted","Data":"c3af3387de315c5c66083683af2f4783c7817f857cb668f4dc3e4e14832878b9"}
Oct 30 20:20:11.246082 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:11.246032    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" podStartSLOduration=1.246018679 podStartE2EDuration="1.246018679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:11.244573161 +0000 UTC m=+3858.455790305" watchObservedRunningTime="2026-10-30 20:20:11.246018679 +0000 UTC m=+3858.457235824"
Oct 30 20:20:11.627040 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:11.627012    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:12.136469 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:12.136434    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" event={"ID":"27330f56-6fc8-4ff7-b32c-424e5330af41","Type":"ContainerStarted","Data":"f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc"}
Oct 30 20:20:12.631624 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:12.631599    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:13.140998 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:13.140975    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:13.141919 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:13.141897    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:19.830888 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:19.830842    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"]
Oct 30 20:20:19.834540 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:19.834521    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:19.935855 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:19.935801    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-14ab4d-serving-cert\""
Oct 30 20:20:19.935855 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:19.935832    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8c9cv\""
Oct 30 20:20:19.936098 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:19.935889    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:20:19.938446 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:19.938420    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"]
Oct 30 20:20:20.026201 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.026172    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.026352 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.026208    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crdr\" (UniqueName: \"kubernetes.io/projected/a618ba8c-4fd2-433b-a205-429cbf0d4029-kube-api-access-2crdr\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.026352 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.026309    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a618ba8c-4fd2-433b-a205-429cbf0d4029-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.026431 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.026362    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-metrics-certs\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.127852 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.127752    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a618ba8c-4fd2-433b-a205-429cbf0d4029-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.127852 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.127807    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-metrics-certs\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.128049 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.127947    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.128049 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.127982    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crdr\" (UniqueName: \"kubernetes.io/projected/a618ba8c-4fd2-433b-a205-429cbf0d4029-kube-api-access-2crdr\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.128548 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.128523    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a618ba8c-4fd2-433b-a205-429cbf0d4029-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.130509 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.130487    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-metrics-certs\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.130635 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.130612    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.184418 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.184389    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crdr\" (UniqueName: \"kubernetes.io/projected/a618ba8c-4fd2-433b-a205-429cbf0d4029-kube-api-access-2crdr\") pod \"maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.448725 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.448682    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:20.817593 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.817535    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"]
Oct 30 20:20:20.819943 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:20:20.819920    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda618ba8c_4fd2_433b_a205_429cbf0d4029.slice/crio-dca41706560317ccc55ebc044ba469a06fda9f12b97d8470ca4853a7c81bcd85 WatchSource:0}: Error finding container dca41706560317ccc55ebc044ba469a06fda9f12b97d8470ca4853a7c81bcd85: Status 404 returned error can't find the container with id dca41706560317ccc55ebc044ba469a06fda9f12b97d8470ca4853a7c81bcd85
Oct 30 20:20:20.821502 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:20.821485    4420 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 20:20:21.178441 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:21.178414    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9" event={"ID":"a618ba8c-4fd2-433b-a205-429cbf0d4029","Type":"ContainerStarted","Data":"dca41706560317ccc55ebc044ba469a06fda9f12b97d8470ca4853a7c81bcd85"}
Oct 30 20:20:22.184680 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:22.184640    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9" event={"ID":"a618ba8c-4fd2-433b-a205-429cbf0d4029","Type":"ContainerStarted","Data":"41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81"}
Oct 30 20:20:22.185056 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:22.184733    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:22.298479 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:22.298430    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9" podStartSLOduration=2.420469563 podStartE2EDuration="3.298417199s" totalImagesPullingTime="877.947636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:22.297142251 +0000 UTC m=+3869.508359397" watchObservedRunningTime="2026-10-30 20:20:22.298417199 +0000 UTC m=+3869.509634344"
Oct 30 20:20:28.195612 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:28.195579    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:20:35.195461 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:20:35.195416    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:20:36.252077 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:36.252053    4420 generic.go:356] "Generic (PLEG): container finished" podID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerID="f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc" exitCode=0
Oct 30 20:20:36.252406 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:36.252119    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" event={"ID":"27330f56-6fc8-4ff7-b32c-424e5330af41","Type":"ContainerDied","Data":"f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc"}
Oct 30 20:20:36.252448 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:36.252436    4420 scope.go:122] "RemoveContainer" containerID="f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc"
Oct 30 20:20:36.253478 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:36.253454    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:20:36.253530 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:36.253520    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:20:36.253572 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:36.253551    4420 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"}
Oct 30 20:20:37.259201 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:37.259159    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" event={"ID":"27330f56-6fc8-4ff7-b32c-424e5330af41","Type":"ContainerStarted","Data":"96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda"}
Oct 30 20:20:37.627437 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:37.627347    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:37.627642 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:37.627620    4420 patch_prober.go:29] interesting pod/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:20:37.627701 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:37.627668    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 20:20:38.627196 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:38.627159    4420 patch_prober.go:29] interesting pod/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:20:38.627578 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:38.627214    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 20:20:39.627734 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:39.627697    4420 patch_prober.go:29] interesting pod/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:20:39.628156 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:39.627753    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 20:20:40.628054 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:40.628023    4420 patch_prober.go:29] interesting pod/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:20:40.628446 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:40.628073    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 20:20:41.627503 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:41.627466    4420 patch_prober.go:29] interesting pod/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 20:20:41.627704 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:41.627541    4420 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 20:20:42.630996 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:42.630969    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:42.631437 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:42.631385    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:42.632118 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:42.632100    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:20:54.079558 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:20:54.079522    4420 scope.go:122] "RemoveContainer" containerID="bd36dfb8c0286bea428803abdceea1a85c0bfb132bdb95768c28e1067db180a6"
Oct 30 20:21:05.194191 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:21:05.194108    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:21:35.196716 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:21:35.196668    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:21:53.233749 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:53.233713    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"]
Oct 30 20:21:53.234331 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:53.233959    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9" podUID="a618ba8c-4fd2-433b-a205-429cbf0d4029" containerName="maas-api" containerID="cri-o://41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81" gracePeriod=30
Oct 30 20:21:54.504007 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.503977    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:21:54.585509 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.585453    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a618ba8c-4fd2-433b-a205-429cbf0d4029-kube-api-access-2crdr\" (UniqueName: \"kubernetes.io/projected/a618ba8c-4fd2-433b-a205-429cbf0d4029-kube-api-access-2crdr\") pod \"a618ba8c-4fd2-433b-a205-429cbf0d4029\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") "
Oct 30 20:21:54.585509 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.585501    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-metrics-certs\") pod \"a618ba8c-4fd2-433b-a205-429cbf0d4029\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") "
Oct 30 20:21:54.585663 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.585522    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-maas-api-tls\") pod \"a618ba8c-4fd2-433b-a205-429cbf0d4029\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") "
Oct 30 20:21:54.585663 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.585548    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a618ba8c-4fd2-433b-a205-429cbf0d4029-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a618ba8c-4fd2-433b-a205-429cbf0d4029-openshift-service-ca\") pod \"a618ba8c-4fd2-433b-a205-429cbf0d4029\" (UID: \"a618ba8c-4fd2-433b-a205-429cbf0d4029\") "
Oct 30 20:21:54.585973 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.585944    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a618ba8c-4fd2-433b-a205-429cbf0d4029-openshift-service-ca" pod "a618ba8c-4fd2-433b-a205-429cbf0d4029" (UID: "a618ba8c-4fd2-433b-a205-429cbf0d4029"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:21:54.587661 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.587640    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-metrics-certs" pod "a618ba8c-4fd2-433b-a205-429cbf0d4029" (UID: "a618ba8c-4fd2-433b-a205-429cbf0d4029"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:54.587753 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.587660    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-maas-api-tls" pod "a618ba8c-4fd2-433b-a205-429cbf0d4029" (UID: "a618ba8c-4fd2-433b-a205-429cbf0d4029"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:21:54.587753 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.587667    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a618ba8c-4fd2-433b-a205-429cbf0d4029-kube-api-access-2crdr" pod "a618ba8c-4fd2-433b-a205-429cbf0d4029" (UID: "a618ba8c-4fd2-433b-a205-429cbf0d4029"). InnerVolumeSpecName "kube-api-access-2crdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:21:54.665024 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.664995    4420 generic.go:356] "Generic (PLEG): container finished" podID="a618ba8c-4fd2-433b-a205-429cbf0d4029" containerID="41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81" exitCode=0
Oct 30 20:21:54.665164 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.665037    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9" event={"ID":"a618ba8c-4fd2-433b-a205-429cbf0d4029","Type":"ContainerDied","Data":"41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81"}
Oct 30 20:21:54.665164 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.665051    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"
Oct 30 20:21:54.665164 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.665076    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9" event={"ID":"a618ba8c-4fd2-433b-a205-429cbf0d4029","Type":"ContainerDied","Data":"dca41706560317ccc55ebc044ba469a06fda9f12b97d8470ca4853a7c81bcd85"}
Oct 30 20:21:54.665164 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.665099    4420 scope.go:122] "RemoveContainer" containerID="41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81"
Oct 30 20:21:54.677716 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.675868    4420 scope.go:122] "RemoveContainer" containerID="41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81"
Oct 30 20:21:54.677716 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:21:54.676245    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81\": container with ID starting with 41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81 not found: ID does not exist" containerID="41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81"
Oct 30 20:21:54.677716 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.676281    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81"} err="failed to get container status \"41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81\": rpc error: code = NotFound desc = could not find container \"41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81\": container with ID starting with 41393cd4a623e3d7d6b91366e854ea924539481bf05fb78b5473d85921992d81 not found: ID does not exist"
Oct 30 20:21:54.686543 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.686521    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2crdr\" (UniqueName: \"kubernetes.io/projected/a618ba8c-4fd2-433b-a205-429cbf0d4029-kube-api-access-2crdr\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:54.686635 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.686545    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:54.686635 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.686555    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a618ba8c-4fd2-433b-a205-429cbf0d4029-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:54.686635 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.686563    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a618ba8c-4fd2-433b-a205-429cbf0d4029-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:21:54.826970 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.826937    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"]
Oct 30 20:21:54.832782 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.832764    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-14ab4d-68d9b56b6c-ch9b9"]
Oct 30 20:21:54.894577 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:21:54.894552    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a618ba8c-4fd2-433b-a205-429cbf0d4029" path="/var/lib/kubelet/pods/a618ba8c-4fd2-433b-a205-429cbf0d4029/volumes"
Oct 30 20:22:02.973805 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:02.973770    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"]
Oct 30 20:22:02.974225 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:02.973989    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy" containerID="cri-o://96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda" gracePeriod=30
Oct 30 20:22:05.195692 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:22:05.195658    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:22:08.047563 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.047534    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"]
Oct 30 20:22:08.096490 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.096420    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xw5rq"]
Oct 30 20:22:08.224515 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.224486    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:22:08.301755 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.301706    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-kube-api-access-mfx6j\" (UniqueName: \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-kube-api-access-mfx6j\") pod \"27330f56-6fc8-4ff7-b32c-424e5330af41\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") "
Oct 30 20:22:08.301755 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.301741    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27330f56-6fc8-4ff7-b32c-424e5330af41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27330f56-6fc8-4ff7-b32c-424e5330af41-istiod-ca-cert\") pod \"27330f56-6fc8-4ff7-b32c-424e5330af41\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") "
Oct 30 20:22:08.302030 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.301763    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-envoy\") pod \"27330f56-6fc8-4ff7-b32c-424e5330af41\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") "
Oct 30 20:22:08.302030 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.301790    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-podinfo\") pod \"27330f56-6fc8-4ff7-b32c-424e5330af41\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") "
Oct 30 20:22:08.302030 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.301833    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-credential-socket\") pod \"27330f56-6fc8-4ff7-b32c-424e5330af41\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") "
Oct 30 20:22:08.302030 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.301946    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-socket\") pod \"27330f56-6fc8-4ff7-b32c-424e5330af41\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") "
Oct 30 20:22:08.302272 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302081    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-token\" (UniqueName: \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-token\") pod \"27330f56-6fc8-4ff7-b32c-424e5330af41\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") "
Oct 30 20:22:08.302272 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302105    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-credential-socket" pod "27330f56-6fc8-4ff7-b32c-424e5330af41" (UID: "27330f56-6fc8-4ff7-b32c-424e5330af41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:08.302272 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302187    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-data\") pod \"27330f56-6fc8-4ff7-b32c-424e5330af41\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") "
Oct 30 20:22:08.302272 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302195    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27330f56-6fc8-4ff7-b32c-424e5330af41-istiod-ca-cert" pod "27330f56-6fc8-4ff7-b32c-424e5330af41" (UID: "27330f56-6fc8-4ff7-b32c-424e5330af41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:22:08.302272 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302222    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-certs\") pod \"27330f56-6fc8-4ff7-b32c-424e5330af41\" (UID: \"27330f56-6fc8-4ff7-b32c-424e5330af41\") "
Oct 30 20:22:08.302570 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302299    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-socket" pod "27330f56-6fc8-4ff7-b32c-424e5330af41" (UID: "27330f56-6fc8-4ff7-b32c-424e5330af41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:08.302570 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302477    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-data" pod "27330f56-6fc8-4ff7-b32c-424e5330af41" (UID: "27330f56-6fc8-4ff7-b32c-424e5330af41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:08.302861 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302707    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27330f56-6fc8-4ff7-b32c-424e5330af41-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:08.302861 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302728    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:08.302861 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302730    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-certs" pod "27330f56-6fc8-4ff7-b32c-424e5330af41" (UID: "27330f56-6fc8-4ff7-b32c-424e5330af41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:08.302861 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302742    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:08.302861 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.302755    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:08.304128 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.304091    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-envoy" pod "27330f56-6fc8-4ff7-b32c-424e5330af41" (UID: "27330f56-6fc8-4ff7-b32c-424e5330af41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:22:08.304224 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.304175    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-kube-api-access-mfx6j" pod "27330f56-6fc8-4ff7-b32c-424e5330af41" (UID: "27330f56-6fc8-4ff7-b32c-424e5330af41"). InnerVolumeSpecName "kube-api-access-mfx6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:08.304271 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.304226    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-podinfo" pod "27330f56-6fc8-4ff7-b32c-424e5330af41" (UID: "27330f56-6fc8-4ff7-b32c-424e5330af41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:22:08.304472 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.304451    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-token" pod "27330f56-6fc8-4ff7-b32c-424e5330af41" (UID: "27330f56-6fc8-4ff7-b32c-424e5330af41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:22:08.403963 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.403939    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:08.403963 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.403963    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:08.404121 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.403977    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:08.404121 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.403988    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27330f56-6fc8-4ff7-b32c-424e5330af41-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:08.404121 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.404001    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfx6j\" (UniqueName: \"kubernetes.io/projected/27330f56-6fc8-4ff7-b32c-424e5330af41-kube-api-access-mfx6j\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:22:08.734305 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.734272    4420 generic.go:356] "Generic (PLEG): container finished" podID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerID="96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda" exitCode=0
Oct 30 20:22:08.734546 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.734321    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"
Oct 30 20:22:08.734546 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.734354    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" event={"ID":"27330f56-6fc8-4ff7-b32c-424e5330af41","Type":"ContainerDied","Data":"96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda"}
Oct 30 20:22:08.734546 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.734429    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b" event={"ID":"27330f56-6fc8-4ff7-b32c-424e5330af41","Type":"ContainerDied","Data":"c3af3387de315c5c66083683af2f4783c7817f857cb668f4dc3e4e14832878b9"}
Oct 30 20:22:08.734546 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.734452    4420 scope.go:122] "RemoveContainer" containerID="96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda"
Oct 30 20:22:08.743203 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.743186    4420 scope.go:122] "RemoveContainer" containerID="f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc"
Oct 30 20:22:08.750736 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.750719    4420 scope.go:122] "RemoveContainer" containerID="96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda"
Oct 30 20:22:08.750988 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:22:08.750971    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda\": container with ID starting with 96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda not found: ID does not exist" containerID="96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda"
Oct 30 20:22:08.751052 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.750996    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda"} err="failed to get container status \"96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda\": rpc error: code = NotFound desc = could not find container \"96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda\": container with ID starting with 96777b075f5054981d62af2a88128e36ed86246a0189631798e28ca8fd636cda not found: ID does not exist"
Oct 30 20:22:08.751052 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.751011    4420 scope.go:122] "RemoveContainer" containerID="f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc"
Oct 30 20:22:08.751251 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:22:08.751235    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc\": container with ID starting with f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc not found: ID does not exist" containerID="f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc"
Oct 30 20:22:08.751296 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.751259    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc"} err="failed to get container status \"f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc\": rpc error: code = NotFound desc = could not find container \"f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc\": container with ID starting with f4e66091c4f95a432aade49e6e2f04bfec2369051e57b20cecdcdf577becd6fc not found: ID does not exist"
Oct 30 20:22:08.894328 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.894307    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2043e99f-17e4-4e47-9c74-e5dd7c390277" path="/var/lib/kubelet/pods/2043e99f-17e4-4e47-9c74-e5dd7c390277/volumes"
Oct 30 20:22:08.898433 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.898414    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"]
Oct 30 20:22:08.947448 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:08.947425    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-14ab4d-openshift-default-6854f45b85ck67b"]
Oct 30 20:22:10.893881 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:10.893852    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" path="/var/lib/kubelet/pods/27330f56-6fc8-4ff7-b32c-424e5330af41/volumes"
Oct 30 20:22:34.900670 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:34.900598    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"]
Oct 30 20:22:34.901141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:34.901000    4420 cpu_manager.go:414] "containerMap: removing container" podUID="a618ba8c-4fd2-433b-a205-429cbf0d4029" containerName="maas-api"
Oct 30 20:22:34.901141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:34.901011    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a618ba8c-4fd2-433b-a205-429cbf0d4029" containerName="maas-api"
Oct 30 20:22:34.901141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:34.901028    4420 cpu_manager.go:414] "containerMap: removing container" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy"
Oct 30 20:22:34.901141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:34.901033    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy"
Oct 30 20:22:34.901141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:34.901044    4420 cpu_manager.go:414] "containerMap: removing container" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy"
Oct 30 20:22:34.901141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:34.901049    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy"
Oct 30 20:22:34.901141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:34.901103    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy"
Oct 30 20:22:34.901141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:34.901113    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="27330f56-6fc8-4ff7-b32c-424e5330af41" containerName="istio-proxy"
Oct 30 20:22:34.901141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:34.901122    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="a618ba8c-4fd2-433b-a205-429cbf0d4029" containerName="maas-api"
Oct 30 20:22:34.904120 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:34.904102    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.004755 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.004729    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-68b4cd44-kserve-self-signed-certs\""
Oct 30 20:22:35.006653 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.006633    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"]
Oct 30 20:22:35.027064 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.027035    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b7ns\" (UniqueName: \"kubernetes.io/projected/07d17986-61e3-4702-9c5d-670115375f3e-kube-api-access-4b7ns\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.027170 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.027075    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-home\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.027170 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.027105    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d17986-61e3-4702-9c5d-670115375f3e-tls-certs\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.027258 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.027237    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-tmp-dir\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.027368 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.027322    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-dshm\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.027415 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.027398    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-model-cache\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.128910 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.128883    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d17986-61e3-4702-9c5d-670115375f3e-tls-certs\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.129102 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.128915    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-tmp-dir\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.129102 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.128960    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-dshm\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.129102 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.129003    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-model-cache\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.129102 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.129035    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b7ns\" (UniqueName: \"kubernetes.io/projected/07d17986-61e3-4702-9c5d-670115375f3e-kube-api-access-4b7ns\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.129102 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.129056    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-home\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.129462 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.129436    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-tmp-dir\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.129540 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.129446    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-home\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.129540 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.129497    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-model-cache\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.131380 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.131359    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-dshm\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.131658 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.131638    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d17986-61e3-4702-9c5d-670115375f3e-tls-certs\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.186334 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.186278    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b7ns\" (UniqueName: \"kubernetes.io/projected/07d17986-61e3-4702-9c5d-670115375f3e-kube-api-access-4b7ns\") pod \"e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") " pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.198126 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:22:35.198098    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:22:35.226180 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.226159    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:35.600880 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.600855    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"]
Oct 30 20:22:35.602733 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:22:35.602706    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07d17986_61e3_4702_9c5d_670115375f3e.slice/crio-a45c724a31ff0927be4d81f1ee2d5a46e76462e73b8d9f79b81b70d4c47d9c89 WatchSource:0}: Error finding container a45c724a31ff0927be4d81f1ee2d5a46e76462e73b8d9f79b81b70d4c47d9c89: Status 404 returned error can't find the container with id a45c724a31ff0927be4d81f1ee2d5a46e76462e73b8d9f79b81b70d4c47d9c89
Oct 30 20:22:35.866542 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.866467    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts" event={"ID":"07d17986-61e3-4702-9c5d-670115375f3e","Type":"ContainerStarted","Data":"1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3"}
Oct 30 20:22:35.866542 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.866512    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts" event={"ID":"07d17986-61e3-4702-9c5d-670115375f3e","Type":"ContainerStarted","Data":"a45c724a31ff0927be4d81f1ee2d5a46e76462e73b8d9f79b81b70d4c47d9c89"}
Oct 30 20:22:35.981398 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:35.981352    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts" podStartSLOduration=1.981338793 podStartE2EDuration="1.981338793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:35.979720796 +0000 UTC m=+4003.190937940" watchObservedRunningTime="2026-10-30 20:22:35.981338793 +0000 UTC m=+4003.192555936"
Oct 30 20:22:36.226599 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:36.226574    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:45.226402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:45.226365    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:45.240825 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:45.240783    4420 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:22:45.951938 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:22:45.951896    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:23:05.252541 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:23:05.252502    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:23:06.043971 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:06.043931    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-748lj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:23:06.044225 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:06.043986    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:23:06.557955 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:06.557926    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-8r4gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:23:06.558349 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:06.557988    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" podUID="ca4ca4ee-283d-4613-8d97-09478295784d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:23:31.084651 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.084610    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"]
Oct 30 20:23:31.088143 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.088128    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:31.139761 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.139738    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:23:31.142478 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.142464    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:23:31.156305 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.156281    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nw4z7\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:31.156433 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.156317    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w99p\" (UniqueName: \"kubernetes.io/projected/842cc94e-1bf2-4863-b5da-68e0c081a5e3-kube-api-access-4w99p\") pod \"maas-api-revoke-sub-452b6d7b-nw4z7\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:31.156433 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.156397    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nw4z7\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:31.193908 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.193887    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"]
Oct 30 20:23:31.257403 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.257377    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nw4z7\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:31.257506 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.257416    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w99p\" (UniqueName: \"kubernetes.io/projected/842cc94e-1bf2-4863-b5da-68e0c081a5e3-kube-api-access-4w99p\") pod \"maas-api-revoke-sub-452b6d7b-nw4z7\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:31.257506 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.257461    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nw4z7\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:31.258232 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.258137    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nw4z7\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:31.259869 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.259846    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nw4z7\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:31.313411 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.313381    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w99p\" (UniqueName: \"kubernetes.io/projected/842cc94e-1bf2-4863-b5da-68e0c081a5e3-kube-api-access-4w99p\") pod \"maas-api-revoke-sub-452b6d7b-nw4z7\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:31.400697 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.400669    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:31.769325 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:31.769305    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"]
Oct 30 20:23:31.771602 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:23:31.771575    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842cc94e_1bf2_4863_b5da_68e0c081a5e3.slice/crio-d58466c025d23f2ddac12e93892d69fe15ba6d328318d30ed8b45ac995cb51ea WatchSource:0}: Error finding container d58466c025d23f2ddac12e93892d69fe15ba6d328318d30ed8b45ac995cb51ea: Status 404 returned error can't find the container with id d58466c025d23f2ddac12e93892d69fe15ba6d328318d30ed8b45ac995cb51ea
Oct 30 20:23:32.164903 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:32.164874    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7" event={"ID":"842cc94e-1bf2-4863-b5da-68e0c081a5e3","Type":"ContainerStarted","Data":"7df11fd06b8f6656c9f0cb0ac966b212c738e48dfd04bc21ff717cc1bab4cb69"}
Oct 30 20:23:32.165235 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:32.164951    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7" event={"ID":"842cc94e-1bf2-4863-b5da-68e0c081a5e3","Type":"ContainerStarted","Data":"d58466c025d23f2ddac12e93892d69fe15ba6d328318d30ed8b45ac995cb51ea"}
Oct 30 20:23:32.280491 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:32.280449    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7" podStartSLOduration=1.280436535 podStartE2EDuration="1.280436535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:32.279030987 +0000 UTC m=+4059.490248131" watchObservedRunningTime="2026-10-30 20:23:32.280436535 +0000 UTC m=+4059.491653679"
Oct 30 20:23:33.173693 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:33.173647    4420 generic.go:356] "Generic (PLEG): container finished" podID="842cc94e-1bf2-4863-b5da-68e0c081a5e3" containerID="7df11fd06b8f6656c9f0cb0ac966b212c738e48dfd04bc21ff717cc1bab4cb69" exitCode=0
Oct 30 20:23:33.174140 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:33.173709    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7" event={"ID":"842cc94e-1bf2-4863-b5da-68e0c081a5e3","Type":"ContainerDied","Data":"7df11fd06b8f6656c9f0cb0ac966b212c738e48dfd04bc21ff717cc1bab4cb69"}
Oct 30 20:23:34.588838 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:34.583410    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:34.744452 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:34.744408    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-xks-ca\") pod \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") "
Oct 30 20:23:34.744645 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:34.744498    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-openshift-ca\") pod \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") "
Oct 30 20:23:34.744645 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:34.744631    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/842cc94e-1bf2-4863-b5da-68e0c081a5e3-kube-api-access-4w99p\" (UniqueName: \"kubernetes.io/projected/842cc94e-1bf2-4863-b5da-68e0c081a5e3-kube-api-access-4w99p\") pod \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\" (UID: \"842cc94e-1bf2-4863-b5da-68e0c081a5e3\") "
Oct 30 20:23:34.745355 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:34.745329    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-openshift-ca" pod "842cc94e-1bf2-4863-b5da-68e0c081a5e3" (UID: "842cc94e-1bf2-4863-b5da-68e0c081a5e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:23:34.747676 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:34.747647    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/842cc94e-1bf2-4863-b5da-68e0c081a5e3-kube-api-access-4w99p" pod "842cc94e-1bf2-4863-b5da-68e0c081a5e3" (UID: "842cc94e-1bf2-4863-b5da-68e0c081a5e3"). InnerVolumeSpecName "kube-api-access-4w99p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:34.747784 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:34.747647    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-xks-ca" pod "842cc94e-1bf2-4863-b5da-68e0c081a5e3" (UID: "842cc94e-1bf2-4863-b5da-68e0c081a5e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:34.845508 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:34.845448    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w99p\" (UniqueName: \"kubernetes.io/projected/842cc94e-1bf2-4863-b5da-68e0c081a5e3-kube-api-access-4w99p\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:34.845508 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:34.845472    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:34.845508 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:34.845483    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/842cc94e-1bf2-4863-b5da-68e0c081a5e3-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:35.206377 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:35.206341    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7" event={"ID":"842cc94e-1bf2-4863-b5da-68e0c081a5e3","Type":"ContainerDied","Data":"d58466c025d23f2ddac12e93892d69fe15ba6d328318d30ed8b45ac995cb51ea"}
Oct 30 20:23:35.206626 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:35.206609    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d58466c025d23f2ddac12e93892d69fe15ba6d328318d30ed8b45ac995cb51ea"
Oct 30 20:23:35.206915 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:35.206898    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"
Oct 30 20:23:35.252889 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:23:35.252855    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:23:44.000124 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:44.000089    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"]
Oct 30 20:23:54.121209 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:54.121176    4420 scope.go:122] "RemoveContainer" containerID="3690e501e888c4be8f131dc1649beb7fb9b280a109c646f386896bcf3f94542a"
Oct 30 20:23:59.019532 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.019481    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts" podUID="07d17986-61e3-4702-9c5d-670115375f3e" containerName="main" containerID="cri-o://1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3" gracePeriod=45
Oct 30 20:23:59.281841 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.281755    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:23:59.341746 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.341711    4420 generic.go:356] "Generic (PLEG): container finished" podID="07d17986-61e3-4702-9c5d-670115375f3e" containerID="1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3" exitCode=0
Oct 30 20:23:59.341954 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.341754    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"
Oct 30 20:23:59.341954 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.341780    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts" event={"ID":"07d17986-61e3-4702-9c5d-670115375f3e","Type":"ContainerDied","Data":"1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3"}
Oct 30 20:23:59.341954 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.341831    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts" event={"ID":"07d17986-61e3-4702-9c5d-670115375f3e","Type":"ContainerDied","Data":"a45c724a31ff0927be4d81f1ee2d5a46e76462e73b8d9f79b81b70d4c47d9c89"}
Oct 30 20:23:59.341954 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.341842    4420 scope.go:122] "RemoveContainer" containerID="1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3"
Oct 30 20:23:59.350571 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.350535    4420 scope.go:122] "RemoveContainer" containerID="1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3"
Oct 30 20:23:59.350838 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:23:59.350801    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3\": container with ID starting with 1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3 not found: ID does not exist" containerID="1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3"
Oct 30 20:23:59.350949 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.350844    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3"} err="failed to get container status \"1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3\": rpc error: code = NotFound desc = could not find container \"1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3\": container with ID starting with 1e18f8be5db8b00fd15389b951e7359911733e5ed86c6f345177053d9c52f1a3 not found: ID does not exist"
Oct 30 20:23:59.386140 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386115    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-tmp-dir\") pod \"07d17986-61e3-4702-9c5d-670115375f3e\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") "
Oct 30 20:23:59.386248 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386174    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-home\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-home\") pod \"07d17986-61e3-4702-9c5d-670115375f3e\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") "
Oct 30 20:23:59.386248 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386229    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-model-cache\") pod \"07d17986-61e3-4702-9c5d-670115375f3e\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") "
Oct 30 20:23:59.386361 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386292    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-dshm\") pod \"07d17986-61e3-4702-9c5d-670115375f3e\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") "
Oct 30 20:23:59.386361 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386342    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-tmp-dir" pod "07d17986-61e3-4702-9c5d-670115375f3e" (UID: "07d17986-61e3-4702-9c5d-670115375f3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.386468 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386412    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-home" pod "07d17986-61e3-4702-9c5d-670115375f3e" (UID: "07d17986-61e3-4702-9c5d-670115375f3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.386468 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386454    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07d17986-61e3-4702-9c5d-670115375f3e-tls-certs\" (UniqueName: \"kubernetes.io/secret/07d17986-61e3-4702-9c5d-670115375f3e-tls-certs\") pod \"07d17986-61e3-4702-9c5d-670115375f3e\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") "
Oct 30 20:23:59.386576 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386491    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-model-cache" pod "07d17986-61e3-4702-9c5d-670115375f3e" (UID: "07d17986-61e3-4702-9c5d-670115375f3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.386576 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386493    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d17986-61e3-4702-9c5d-670115375f3e-kube-api-access-4b7ns\" (UniqueName: \"kubernetes.io/projected/07d17986-61e3-4702-9c5d-670115375f3e-kube-api-access-4b7ns\") pod \"07d17986-61e3-4702-9c5d-670115375f3e\" (UID: \"07d17986-61e3-4702-9c5d-670115375f3e\") "
Oct 30 20:23:59.386844 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386808    4420 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-model-cache\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.386844 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386844    4420 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-tmp-dir\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.387173 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.386853    4420 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-home\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.388425 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.388401    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-dshm" pod "07d17986-61e3-4702-9c5d-670115375f3e" (UID: "07d17986-61e3-4702-9c5d-670115375f3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:23:59.388502 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.388486    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d17986-61e3-4702-9c5d-670115375f3e-tls-certs" pod "07d17986-61e3-4702-9c5d-670115375f3e" (UID: "07d17986-61e3-4702-9c5d-670115375f3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:23:59.388645 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.388627    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d17986-61e3-4702-9c5d-670115375f3e-kube-api-access-4b7ns" pod "07d17986-61e3-4702-9c5d-670115375f3e" (UID: "07d17986-61e3-4702-9c5d-670115375f3e"). InnerVolumeSpecName "kube-api-access-4b7ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:23:59.488097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.488077    4420 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d17986-61e3-4702-9c5d-670115375f3e-dshm\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.488097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.488099    4420 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d17986-61e3-4702-9c5d-670115375f3e-tls-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.488242 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.488108    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b7ns\" (UniqueName: \"kubernetes.io/projected/07d17986-61e3-4702-9c5d-670115375f3e-kube-api-access-4b7ns\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:23:59.760660 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.760637    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"]
Oct 30 20:23:59.763865 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:23:59.763843    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-68b4cd44-kserve-6d7c4f4cfc-k7fts"]
Oct 30 20:24:00.894076 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:00.894045    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d17986-61e3-4702-9c5d-670115375f3e" path="/var/lib/kubelet/pods/07d17986-61e3-4702-9c5d-670115375f3e/volumes"
Oct 30 20:24:03.401950 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.401918    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"]
Oct 30 20:24:03.402398 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.402384    4420 cpu_manager.go:414] "containerMap: removing container" podUID="07d17986-61e3-4702-9c5d-670115375f3e" containerName="main"
Oct 30 20:24:03.402449 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.402400    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d17986-61e3-4702-9c5d-670115375f3e" containerName="main"
Oct 30 20:24:03.402449 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.402423    4420 cpu_manager.go:414] "containerMap: removing container" podUID="842cc94e-1bf2-4863-b5da-68e0c081a5e3" containerName="revoke-keys"
Oct 30 20:24:03.402449 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.402428    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="842cc94e-1bf2-4863-b5da-68e0c081a5e3" containerName="revoke-keys"
Oct 30 20:24:03.402565 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.402508    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d17986-61e3-4702-9c5d-670115375f3e" containerName="main"
Oct 30 20:24:03.402565 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.402519    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="842cc94e-1bf2-4863-b5da-68e0c081a5e3" containerName="revoke-keys"
Oct 30 20:24:03.405678 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.405663    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:03.462521 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.462492    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"]
Oct 30 20:24:03.509027 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.508993    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:24:03.509027 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.509016    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:24:03.521586 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.521560    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z67x7\" (UniqueName: \"kubernetes.io/projected/260ce849-76ee-4587-af91-f091706f4b89-kube-api-access-z67x7\") pod \"maas-api-revoke-sub-3e31a506-5hcdc\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:03.521737 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.521604    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260ce849-76ee-4587-af91-f091706f4b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-5hcdc\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:03.521737 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.521631    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260ce849-76ee-4587-af91-f091706f4b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-5hcdc\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:03.622448 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.622425    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z67x7\" (UniqueName: \"kubernetes.io/projected/260ce849-76ee-4587-af91-f091706f4b89-kube-api-access-z67x7\") pod \"maas-api-revoke-sub-3e31a506-5hcdc\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:03.622587 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.622465    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260ce849-76ee-4587-af91-f091706f4b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-5hcdc\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:03.622587 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.622489    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260ce849-76ee-4587-af91-f091706f4b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-5hcdc\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:03.623107 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.623083    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260ce849-76ee-4587-af91-f091706f4b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-5hcdc\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:03.624903 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.624884    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260ce849-76ee-4587-af91-f091706f4b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-5hcdc\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:03.678786 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.678733    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z67x7\" (UniqueName: \"kubernetes.io/projected/260ce849-76ee-4587-af91-f091706f4b89-kube-api-access-z67x7\") pod \"maas-api-revoke-sub-3e31a506-5hcdc\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:03.718604 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:03.718578    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:04.099402 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:04.099360    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"]
Oct 30 20:24:04.101255 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:24:04.101221    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260ce849_76ee_4587_af91_f091706f4b89.slice/crio-bca25ca7687f46decddeb19d3f0c0df3b1f3fb0c5bf0c34ae0e9ac3563c5663f WatchSource:0}: Error finding container bca25ca7687f46decddeb19d3f0c0df3b1f3fb0c5bf0c34ae0e9ac3563c5663f: Status 404 returned error can't find the container with id bca25ca7687f46decddeb19d3f0c0df3b1f3fb0c5bf0c34ae0e9ac3563c5663f
Oct 30 20:24:04.370448 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:04.370416    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc" event={"ID":"260ce849-76ee-4587-af91-f091706f4b89","Type":"ContainerStarted","Data":"bca25ca7687f46decddeb19d3f0c0df3b1f3fb0c5bf0c34ae0e9ac3563c5663f"}
Oct 30 20:24:05.245677 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:24:05.245644    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:24:05.395856 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:05.395689    4420 generic.go:356] "Generic (PLEG): container finished" podID="260ce849-76ee-4587-af91-f091706f4b89" containerID="444550121ea3131bd9095cc2f7cc24917d1019d562057330ff98bb6c3dfbeaa6" exitCode=0
Oct 30 20:24:05.396451 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:05.396091    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc" event={"ID":"260ce849-76ee-4587-af91-f091706f4b89","Type":"ContainerDied","Data":"444550121ea3131bd9095cc2f7cc24917d1019d562057330ff98bb6c3dfbeaa6"}
Oct 30 20:24:06.041947 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.041919    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-748lj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:24:06.042135 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.041968    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:24:06.529696 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.529663    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:06.655200 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.655129    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260ce849-76ee-4587-af91-f091706f4b89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260ce849-76ee-4587-af91-f091706f4b89-maas-api-xks-ca\") pod \"260ce849-76ee-4587-af91-f091706f4b89\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") "
Oct 30 20:24:06.655339 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.655252    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260ce849-76ee-4587-af91-f091706f4b89-kube-api-access-z67x7\" (UniqueName: \"kubernetes.io/projected/260ce849-76ee-4587-af91-f091706f4b89-kube-api-access-z67x7\") pod \"260ce849-76ee-4587-af91-f091706f4b89\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") "
Oct 30 20:24:06.655339 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.655310    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260ce849-76ee-4587-af91-f091706f4b89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260ce849-76ee-4587-af91-f091706f4b89-maas-api-openshift-ca\") pod \"260ce849-76ee-4587-af91-f091706f4b89\" (UID: \"260ce849-76ee-4587-af91-f091706f4b89\") "
Oct 30 20:24:06.656029 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.655999    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260ce849-76ee-4587-af91-f091706f4b89-maas-api-openshift-ca" pod "260ce849-76ee-4587-af91-f091706f4b89" (UID: "260ce849-76ee-4587-af91-f091706f4b89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:06.657791 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.657769    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260ce849-76ee-4587-af91-f091706f4b89-maas-api-xks-ca" pod "260ce849-76ee-4587-af91-f091706f4b89" (UID: "260ce849-76ee-4587-af91-f091706f4b89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:06.658360 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.658341    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260ce849-76ee-4587-af91-f091706f4b89-kube-api-access-z67x7" pod "260ce849-76ee-4587-af91-f091706f4b89" (UID: "260ce849-76ee-4587-af91-f091706f4b89"). InnerVolumeSpecName "kube-api-access-z67x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:06.757259 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.757237    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260ce849-76ee-4587-af91-f091706f4b89-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:06.757353 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.757263    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z67x7\" (UniqueName: \"kubernetes.io/projected/260ce849-76ee-4587-af91-f091706f4b89-kube-api-access-z67x7\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:06.757353 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:06.757277    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260ce849-76ee-4587-af91-f091706f4b89-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:07.406723 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:07.406701    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"
Oct 30 20:24:07.406899 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:07.406731    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc" event={"ID":"260ce849-76ee-4587-af91-f091706f4b89","Type":"ContainerDied","Data":"bca25ca7687f46decddeb19d3f0c0df3b1f3fb0c5bf0c34ae0e9ac3563c5663f"}
Oct 30 20:24:07.406899 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:07.406772    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bca25ca7687f46decddeb19d3f0c0df3b1f3fb0c5bf0c34ae0e9ac3563c5663f"
Oct 30 20:24:22.047124 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:22.047082    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"]
Oct 30 20:24:22.096725 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:22.096695    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-gpcht"]
Oct 30 20:24:22.894920 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:22.894893    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a3f6ef-b26d-4e8e-a142-b9ba15bad540" path="/var/lib/kubelet/pods/66a3f6ef-b26d-4e8e-a142-b9ba15bad540/volumes"
Oct 30 20:24:35.196058 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:24:35.196018    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:24:37.700081 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.700050    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"]
Oct 30 20:24:37.700463 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.700451    4420 cpu_manager.go:414] "containerMap: removing container" podUID="260ce849-76ee-4587-af91-f091706f4b89" containerName="revoke-keys"
Oct 30 20:24:37.700506 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.700464    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260ce849-76ee-4587-af91-f091706f4b89" containerName="revoke-keys"
Oct 30 20:24:37.700562 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.700550    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="260ce849-76ee-4587-af91-f091706f4b89" containerName="revoke-keys"
Oct 30 20:24:37.703751 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.703736    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:37.758540 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.758507    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 20:24:37.760797 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.760774    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"]
Oct 30 20:24:37.807359 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.807335    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nlxp4\""
Oct 30 20:24:37.851518 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.851494    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eea3505-4m4xc\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:37.851638 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.851539    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eea3505-4m4xc\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:37.851686 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.851630    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp2tr\" (UniqueName: \"kubernetes.io/projected/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-kube-api-access-lp2tr\") pod \"maas-api-revoke-sub-3eea3505-4m4xc\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:37.953017 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.952967    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp2tr\" (UniqueName: \"kubernetes.io/projected/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-kube-api-access-lp2tr\") pod \"maas-api-revoke-sub-3eea3505-4m4xc\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:37.953137 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.953056    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eea3505-4m4xc\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:37.953137 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.953084    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eea3505-4m4xc\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:37.953725 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.953708    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3eea3505-4m4xc\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:37.955495 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:37.955480    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3eea3505-4m4xc\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:38.009665 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:38.009638    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp2tr\" (UniqueName: \"kubernetes.io/projected/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-kube-api-access-lp2tr\") pod \"maas-api-revoke-sub-3eea3505-4m4xc\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:38.016489 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:38.016466    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:38.387199 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:38.387172    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"]
Oct 30 20:24:38.388829 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:24:38.388785    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd1beb3_3f4a_4b61_81fb_e63046d05ff2.slice/crio-5aeb2d1502ff5c1843fd815a89a0ac3bd50e343b42c2cd1b36b378a1be941d9f WatchSource:0}: Error finding container 5aeb2d1502ff5c1843fd815a89a0ac3bd50e343b42c2cd1b36b378a1be941d9f: Status 404 returned error can't find the container with id 5aeb2d1502ff5c1843fd815a89a0ac3bd50e343b42c2cd1b36b378a1be941d9f
Oct 30 20:24:38.573025 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:38.572998    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc" event={"ID":"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2","Type":"ContainerStarted","Data":"5aeb2d1502ff5c1843fd815a89a0ac3bd50e343b42c2cd1b36b378a1be941d9f"}
Oct 30 20:24:39.579242 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:39.579173    4420 generic.go:356] "Generic (PLEG): container finished" podID="1cd1beb3-3f4a-4b61-81fb-e63046d05ff2" containerID="964f0e26468c925894ef3ed3b1f31fa6d62e72e19f2bedab2c864d48a6a1d302" exitCode=0
Oct 30 20:24:39.579612 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:39.579259    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc" event={"ID":"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2","Type":"ContainerDied","Data":"964f0e26468c925894ef3ed3b1f31fa6d62e72e19f2bedab2c864d48a6a1d302"}
Oct 30 20:24:40.721101 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:40.721079    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:40.879442 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:40.879368    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-kube-api-access-lp2tr\" (UniqueName: \"kubernetes.io/projected/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-kube-api-access-lp2tr\") pod \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") "
Oct 30 20:24:40.879615 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:40.879483    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-openshift-ca\") pod \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") "
Oct 30 20:24:40.879615 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:40.879519    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-xks-ca\") pod \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\" (UID: \"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2\") "
Oct 30 20:24:40.879880 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:40.879853    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-openshift-ca" pod "1cd1beb3-3f4a-4b61-81fb-e63046d05ff2" (UID: "1cd1beb3-3f4a-4b61-81fb-e63046d05ff2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:24:40.881590 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:40.881569    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-xks-ca" pod "1cd1beb3-3f4a-4b61-81fb-e63046d05ff2" (UID: "1cd1beb3-3f4a-4b61-81fb-e63046d05ff2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:24:40.881662 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:40.881611    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-kube-api-access-lp2tr" pod "1cd1beb3-3f4a-4b61-81fb-e63046d05ff2" (UID: "1cd1beb3-3f4a-4b61-81fb-e63046d05ff2"). InnerVolumeSpecName "kube-api-access-lp2tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:24:40.981066 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:40.981039    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-openshift-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:40.981066 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:40.981065    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-maas-api-xks-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:40.981210 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:40.981081    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp2tr\" (UniqueName: \"kubernetes.io/projected/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2-kube-api-access-lp2tr\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:24:41.591180 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:41.591158    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"
Oct 30 20:24:41.591180 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:41.591164    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc" event={"ID":"1cd1beb3-3f4a-4b61-81fb-e63046d05ff2","Type":"ContainerDied","Data":"5aeb2d1502ff5c1843fd815a89a0ac3bd50e343b42c2cd1b36b378a1be941d9f"}
Oct 30 20:24:41.591363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:41.591215    4420 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aeb2d1502ff5c1843fd815a89a0ac3bd50e343b42c2cd1b36b378a1be941d9f"
Oct 30 20:24:51.019223 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.019186    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"]
Oct 30 20:24:51.019700 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.019599    4420 cpu_manager.go:414] "containerMap: removing container" podUID="1cd1beb3-3f4a-4b61-81fb-e63046d05ff2" containerName="revoke-keys"
Oct 30 20:24:51.019700 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.019609    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd1beb3-3f4a-4b61-81fb-e63046d05ff2" containerName="revoke-keys"
Oct 30 20:24:51.019700 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.019665    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd1beb3-3f4a-4b61-81fb-e63046d05ff2" containerName="revoke-keys"
Oct 30 20:24:51.022540 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.022518    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.048106 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.048080    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-wt95w"]
Oct 30 20:24:51.052243 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.052223    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.060557 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.060532    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff00ea5-6209-447f-abf9-4399d463d829-maas-api-tls\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.060688 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.060583    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e4275e-c435-4e96-928f-7069c3b2d4ba-openshift-service-ca\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.060743 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.060678    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-maas-api-tls\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.060743 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.060724    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff00ea5-6209-447f-abf9-4399d463d829-metrics-certs\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.060849 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.060745    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xkr7\" (UniqueName: \"kubernetes.io/projected/6ff00ea5-6209-447f-abf9-4399d463d829-kube-api-access-9xkr7\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.060932 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.060907    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-metrics-certs\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.061001 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.060982    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff00ea5-6209-447f-abf9-4399d463d829-openshift-service-ca\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.061069 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.061055    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qmq6\" (UniqueName: \"kubernetes.io/projected/e1e4275e-c435-4e96-928f-7069c3b2d4ba-kube-api-access-6qmq6\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.095314 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.095296    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 20:24:51.095314 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.095297    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 20:24:51.123648 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.123626    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8c9cv\""
Oct 30 20:24:51.126735 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.126714    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"]
Oct 30 20:24:51.131780 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.131756    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-wt95w"]
Oct 30 20:24:51.161537 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.161511    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff00ea5-6209-447f-abf9-4399d463d829-maas-api-tls\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.161650 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.161548    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e4275e-c435-4e96-928f-7069c3b2d4ba-openshift-service-ca\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.161650 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.161579    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-maas-api-tls\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.161650 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.161622    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff00ea5-6209-447f-abf9-4399d463d829-metrics-certs\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.161841 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.161649    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xkr7\" (UniqueName: \"kubernetes.io/projected/6ff00ea5-6209-447f-abf9-4399d463d829-kube-api-access-9xkr7\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.161841 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.161683    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-metrics-certs\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.161961 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.161843    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff00ea5-6209-447f-abf9-4399d463d829-openshift-service-ca\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.161961 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.161931    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qmq6\" (UniqueName: \"kubernetes.io/projected/e1e4275e-c435-4e96-928f-7069c3b2d4ba-kube-api-access-6qmq6\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.162330 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.162303    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e4275e-c435-4e96-928f-7069c3b2d4ba-openshift-service-ca\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.162481 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.162463    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff00ea5-6209-447f-abf9-4399d463d829-openshift-service-ca\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.164159 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.164141    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff00ea5-6209-447f-abf9-4399d463d829-metrics-certs\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.164550 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.164529    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-metrics-certs\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.164626 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.164562    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-maas-api-tls\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.164665 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.164639    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ff00ea5-6209-447f-abf9-4399d463d829-maas-api-tls\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.218119 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.218100    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xkr7\" (UniqueName: \"kubernetes.io/projected/6ff00ea5-6209-447f-abf9-4399d463d829-kube-api-access-9xkr7\") pod \"maas-api-595cc648c-wt95w\" (UID: \"6ff00ea5-6209-447f-abf9-4399d463d829\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.218456 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.218435    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qmq6\" (UniqueName: \"kubernetes.io/projected/e1e4275e-c435-4e96-928f-7069c3b2d4ba-kube-api-access-6qmq6\") pod \"maas-api-5c48877c75-xv79v\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.335786 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.335737    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:51.364379 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.364358    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:51.710979 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.710953    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"]
Oct 30 20:24:51.712601 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:24:51.712577    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e4275e_c435_4e96_928f_7069c3b2d4ba.slice/crio-aaf5259659a7d0b5df8b3f5969363b200cbfc28aa827cda7e5ff9318ea9078e2 WatchSource:0}: Error finding container aaf5259659a7d0b5df8b3f5969363b200cbfc28aa827cda7e5ff9318ea9078e2: Status 404 returned error can't find the container with id aaf5259659a7d0b5df8b3f5969363b200cbfc28aa827cda7e5ff9318ea9078e2
Oct 30 20:24:51.761396 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:51.761367    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-wt95w"]
Oct 30 20:24:51.762519 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:24:51.762499    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff00ea5_6209_447f_abf9_4399d463d829.slice/crio-8673f4f89d3cc6573ed03c8df2540c0bea28369a0c069d43c610680205f48095 WatchSource:0}: Error finding container 8673f4f89d3cc6573ed03c8df2540c0bea28369a0c069d43c610680205f48095: Status 404 returned error can't find the container with id 8673f4f89d3cc6573ed03c8df2540c0bea28369a0c069d43c610680205f48095
Oct 30 20:24:52.648877 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:52.648842    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v" event={"ID":"e1e4275e-c435-4e96-928f-7069c3b2d4ba","Type":"ContainerStarted","Data":"aaf5259659a7d0b5df8b3f5969363b200cbfc28aa827cda7e5ff9318ea9078e2"}
Oct 30 20:24:52.649933 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:52.649910    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w" event={"ID":"6ff00ea5-6209-447f-abf9-4399d463d829","Type":"ContainerStarted","Data":"8673f4f89d3cc6573ed03c8df2540c0bea28369a0c069d43c610680205f48095"}
Oct 30 20:24:53.656607 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:53.656570    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v" event={"ID":"e1e4275e-c435-4e96-928f-7069c3b2d4ba","Type":"ContainerStarted","Data":"7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7"}
Oct 30 20:24:53.657186 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:53.656631    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:53.658123 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:53.658094    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w" event={"ID":"6ff00ea5-6209-447f-abf9-4399d463d829","Type":"ContainerStarted","Data":"ff509118faa61d9972b4ad077e4361bed4b2c1ed6a251d74c426209012510167"}
Oct 30 20:24:53.658227 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:53.658146    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:53.771418 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:53.771373    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v" podStartSLOduration=2.797663417 podStartE2EDuration="3.771359303s" totalImagesPullingTime="973.695886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:53.769192832 +0000 UTC m=+4140.980409979" watchObservedRunningTime="2026-10-30 20:24:53.771359303 +0000 UTC m=+4140.982576447"
Oct 30 20:24:53.886660 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:53.886616    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w" podStartSLOduration=2.958759349 podStartE2EDuration="3.886604274s" totalImagesPullingTime="927.844925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:24:53.884411732 +0000 UTC m=+4141.095628879" watchObservedRunningTime="2026-10-30 20:24:53.886604274 +0000 UTC m=+4141.097821418"
Oct 30 20:24:59.667985 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:59.667959    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:24:59.668441 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:59.668426    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-wt95w"
Oct 30 20:24:59.928377 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:59.928302    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"]
Oct 30 20:24:59.928567 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:24:59.928519    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v" podUID="e1e4275e-c435-4e96-928f-7069c3b2d4ba" containerName="maas-api" containerID="cri-o://7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7" gracePeriod=30
Oct 30 20:25:00.183639 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.183585    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:25:00.242097 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.242073    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e4275e-c435-4e96-928f-7069c3b2d4ba-kube-api-access-6qmq6\" (UniqueName: \"kubernetes.io/projected/e1e4275e-c435-4e96-928f-7069c3b2d4ba-kube-api-access-6qmq6\") pod \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") "
Oct 30 20:25:00.242240 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.242122    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-metrics-certs\") pod \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") "
Oct 30 20:25:00.242240 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.242152    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-maas-api-tls\") pod \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") "
Oct 30 20:25:00.242333 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.242269    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e4275e-c435-4e96-928f-7069c3b2d4ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e4275e-c435-4e96-928f-7069c3b2d4ba-openshift-service-ca\") pod \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\" (UID: \"e1e4275e-c435-4e96-928f-7069c3b2d4ba\") "
Oct 30 20:25:00.242621 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.242599    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e4275e-c435-4e96-928f-7069c3b2d4ba-openshift-service-ca" pod "e1e4275e-c435-4e96-928f-7069c3b2d4ba" (UID: "e1e4275e-c435-4e96-928f-7069c3b2d4ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:25:00.244238 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.244200    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-maas-api-tls" pod "e1e4275e-c435-4e96-928f-7069c3b2d4ba" (UID: "e1e4275e-c435-4e96-928f-7069c3b2d4ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:00.244329 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.244279    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-metrics-certs" pod "e1e4275e-c435-4e96-928f-7069c3b2d4ba" (UID: "e1e4275e-c435-4e96-928f-7069c3b2d4ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 20:25:00.244329 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.244316    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e4275e-c435-4e96-928f-7069c3b2d4ba-kube-api-access-6qmq6" pod "e1e4275e-c435-4e96-928f-7069c3b2d4ba" (UID: "e1e4275e-c435-4e96-928f-7069c3b2d4ba"). InnerVolumeSpecName "kube-api-access-6qmq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:25:00.343447 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.343423    4420 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-metrics-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:00.343447 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.343444    4420 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e4275e-c435-4e96-928f-7069c3b2d4ba-maas-api-tls\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:00.343637 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.343453    4420 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e4275e-c435-4e96-928f-7069c3b2d4ba-openshift-service-ca\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:00.343637 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.343463    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qmq6\" (UniqueName: \"kubernetes.io/projected/e1e4275e-c435-4e96-928f-7069c3b2d4ba-kube-api-access-6qmq6\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:25:00.691703 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.691672    4420 generic.go:356] "Generic (PLEG): container finished" podID="e1e4275e-c435-4e96-928f-7069c3b2d4ba" containerID="7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7" exitCode=0
Oct 30 20:25:00.692215 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.691712    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v" event={"ID":"e1e4275e-c435-4e96-928f-7069c3b2d4ba","Type":"ContainerDied","Data":"7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7"}
Oct 30 20:25:00.692215 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.691728    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"
Oct 30 20:25:00.692215 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.691759    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-xv79v" event={"ID":"e1e4275e-c435-4e96-928f-7069c3b2d4ba","Type":"ContainerDied","Data":"aaf5259659a7d0b5df8b3f5969363b200cbfc28aa827cda7e5ff9318ea9078e2"}
Oct 30 20:25:00.692215 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.691780    4420 scope.go:122] "RemoveContainer" containerID="7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7"
Oct 30 20:25:00.702141 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.702121    4420 scope.go:122] "RemoveContainer" containerID="7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7"
Oct 30 20:25:00.702407 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:25:00.702386    4420 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7\": container with ID starting with 7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7 not found: ID does not exist" containerID="7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7"
Oct 30 20:25:00.702494 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.702420    4420 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7"} err="failed to get container status \"7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7\": rpc error: code = NotFound desc = could not find container \"7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7\": container with ID starting with 7e34ef103476e5a4d419ed18eef415df213330115aae0ec7151704878e18aef7 not found: ID does not exist"
Oct 30 20:25:00.853527 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.853499    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"]
Oct 30 20:25:00.856906 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.856885    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-xv79v"]
Oct 30 20:25:00.894866 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:00.894841    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e4275e-c435-4e96-928f-7069c3b2d4ba" path="/var/lib/kubelet/pods/e1e4275e-c435-4e96-928f-7069c3b2d4ba/volumes"
Oct 30 20:25:02.047264 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:02.047225    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"]
Oct 30 20:25:02.095798 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:02.095770    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-88pbv"]
Oct 30 20:25:02.925866 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:02.925806    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e58cd73-c383-435d-a14b-0dc45b19afad" path="/var/lib/kubelet/pods/7e58cd73-c383-435d-a14b-0dc45b19afad/volumes"
Oct 30 20:25:05.194952 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:25:05.194921    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:25:15.247990 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:15.247903    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cxf6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:25:15.247990 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:15.247953    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podUID="94d386d8-73a7-4a53-8fec-2d70fcd9a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:25:35.195433 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:25:35.195358    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:25:45.242953 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:45.242910    4420 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-cxf6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 20:25:45.245396 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:45.242984    4420 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-cxf6d" podUID="94d386d8-73a7-4a53-8fec-2d70fcd9a2ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 20:25:54.151825 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:25:54.151780    4420 scope.go:122] "RemoveContainer" containerID="0e10373375f17e1084213c5a59b82ca9fba41b905c027ca3173e8569ff9512f1"
Oct 30 20:26:05.195249 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:26:05.195210    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:26:35.195529 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:26:35.195491    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:26:52.891685 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:26:52.891650    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5d5b686b-r9n2j_24780d87-6614-4bf9-a587-89e043a10f55/manager/0.log"
Oct 30 20:26:54.175134 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:26:54.175106    4420 scope.go:122] "RemoveContainer" containerID="87bac46805cc6bc076f08d7c96dbc5423e74a82b33427cc213d864bcfc59d16b"
Oct 30 20:26:55.288517 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:26:55.288487    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-xr2np_d23d6591-7ebf-431f-9dfd-408a973d4a03/copy-manifests/0.log"
Oct 30 20:26:55.363981 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:26:55.363957    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-xr2np_d23d6591-7ebf-431f-9dfd-408a973d4a03/manager/0.log"
Oct 30 20:26:55.778328 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:26:55.778301    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-s4vzh_5f627072-bcb8-4998-8cb0-51c3e4ead9ac/proxy/0.log"
Oct 30 20:26:57.242478 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:26:57.242449    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-wt95w_6ff00ea5-6209-447f-abf9-4399d463d829/maas-api/0.log"
Oct 30 20:26:57.743496 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:26:57.743465    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-5hcdc_260ce849-76ee-4587-af91-f091706f4b89/revoke-keys/0.log"
Oct 30 20:26:57.958061 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:26:57.958040    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3eea3505-4m4xc_1cd1beb3-3f4a-4b61-81fb-e63046d05ff2/revoke-keys/0.log"
Oct 30 20:26:58.177188 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:26:58.177167    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-nw4z7_842cc94e-1bf2-4863-b5da-68e0c081a5e3/revoke-keys/0.log"
Oct 30 20:26:58.851076 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:26:58.851044    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-h9nq6_7022b85c-4fdf-4e78-bfda-a7d9ce4a0572/postgres/0.log"
Oct 30 20:27:00.878827 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:27:00.878789    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-2hxh9_abce33e0-e121-4270-9869-6ffd925ec08c/manager/0.log"
Oct 30 20:27:01.200681 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:27:01.200641    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-9c76g_95ba72ee-301b-4265-b57a-013c9f16d8b5/manager/0.log"
Oct 30 20:27:05.194865 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:27:05.194771    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:27:14.022951 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:27:14.022917    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6998b7b5-qbhn2_7a3667f5-8616-4e54-ba3e-f9a530059639/kube-auth-proxy/0.log"
Oct 30 20:27:14.479456 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:27:14.479430    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-748lj_442ef4a0-7909-42c2-8567-8d7555361aee/istio-proxy/0.log"
Oct 30 20:27:14.817075 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:27:14.817002    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-8r4gb_ca4ca4ee-283d-4613-8d97-09478295784d/istio-proxy/0.log"
Oct 30 20:27:15.103620 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:27:15.103559    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-cxf6d_94d386d8-73a7-4a53-8fec-2d70fcd9a2ef/istio-proxy/0.log"
Oct 30 20:27:28.791933 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:27:28.791894    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5_e13895f0-43af-4669-8830-40b9c80f2a9b/main/0.log"
Oct 30 20:27:29.339267 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:27:29.339241    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7pz2b_bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4/main/0.log"
Oct 30 20:27:29.901027 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:27:29.900998    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4_593f49af-9c5d-4813-851f-2e1481f27476/main/0.log"
Oct 30 20:27:30.184485 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:27:30.184421    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4_958d4afd-3115-469c-aa3f-ed96a8233238/main/0.log"
Oct 30 20:27:35.196851 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:27:35.196793    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:28:02.079264 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:02.079232    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5d5b686b-r9n2j_24780d87-6614-4bf9-a587-89e043a10f55/manager/0.log"
Oct 30 20:28:05.195352 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:28:05.195317    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:28:14.090325 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:14.090291    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-xr2np_d23d6591-7ebf-431f-9dfd-408a973d4a03/copy-manifests/0.log"
Oct 30 20:28:14.169327 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:14.169304    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69bfcc8945-xr2np_d23d6591-7ebf-431f-9dfd-408a973d4a03/manager/0.log"
Oct 30 20:28:14.602720 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:14.602698    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-7d94f55f4b-s4vzh_5f627072-bcb8-4998-8cb0-51c3e4ead9ac/proxy/0.log"
Oct 30 20:28:15.564920 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:15.564893    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-wt95w_6ff00ea5-6209-447f-abf9-4399d463d829/maas-api/0.log"
Oct 30 20:28:16.075254 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:16.075223    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-5hcdc_260ce849-76ee-4587-af91-f091706f4b89/revoke-keys/0.log"
Oct 30 20:28:16.322515 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:16.322491    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3eea3505-4m4xc_1cd1beb3-3f4a-4b61-81fb-e63046d05ff2/revoke-keys/0.log"
Oct 30 20:28:16.543011 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:16.542987    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-nw4z7_842cc94e-1bf2-4863-b5da-68e0c081a5e3/revoke-keys/0.log"
Oct 30 20:28:17.228428 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:17.228400    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-h9nq6_7022b85c-4fdf-4e78-bfda-a7d9ce4a0572/postgres/0.log"
Oct 30 20:28:28.948203 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:28.948163    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-2hxh9_abce33e0-e121-4270-9869-6ffd925ec08c/manager/0.log"
Oct 30 20:28:29.256999 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:29.256933    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65bfd79598-9c76g_95ba72ee-301b-4265-b57a-013c9f16d8b5/manager/0.log"
Oct 30 20:28:31.679792 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:31.679758    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6998b7b5-qbhn2_7a3667f5-8616-4e54-ba3e-f9a530059639/kube-auth-proxy/0.log"
Oct 30 20:28:32.133936 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:32.133871    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-748lj_442ef4a0-7909-42c2-8567-8d7555361aee/istio-proxy/0.log"
Oct 30 20:28:32.415790 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:32.415763    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-8r4gb_ca4ca4ee-283d-4613-8d97-09478295784d/istio-proxy/0.log"
Oct 30 20:28:32.711193 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:32.711129    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-cxf6d_94d386d8-73a7-4a53-8fec-2d70fcd9a2ef/istio-proxy/0.log"
Oct 30 20:28:35.051678 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:35.051590    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"]
Oct 30 20:28:35.099061 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:35.099032    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nw4z7"]
Oct 30 20:28:35.194791 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:28:35.194760    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:28:36.021289 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:36.021260    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-zh2b5_e13895f0-43af-4669-8830-40b9c80f2a9b/main/0.log"
Oct 30 20:28:36.578960 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:36.578932    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7pz2b_bfd70a6e-d10d-43bf-9b27-1d9b7e7334f4/main/0.log"
Oct 30 20:28:36.895167 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:36.895139    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="842cc94e-1bf2-4863-b5da-68e0c081a5e3" path="/var/lib/kubelet/pods/842cc94e-1bf2-4863-b5da-68e0c081a5e3/volumes"
Oct 30 20:28:37.116121 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:37.116093    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8zddl4_593f49af-9c5d-4813-851f-2e1481f27476/main/0.log"
Oct 30 20:28:37.378578 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:28:37.378504    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-9lff4_958d4afd-3115-469c-aa3f-ed96a8233238/main/0.log"
Oct 30 20:29:05.195297 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:29:05.195258    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:29:07.047033 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:29:07.046999    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"]
Oct 30 20:29:07.095870 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:29:07.095844    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5hcdc"]
Oct 30 20:29:08.894147 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:29:08.894115    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260ce849-76ee-4587-af91-f091706f4b89" path="/var/lib/kubelet/pods/260ce849-76ee-4587-af91-f091706f4b89/volumes"
Oct 30 20:29:27.335671 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:29:27.335646    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5d5b686b-r9n2j_24780d87-6614-4bf9-a587-89e043a10f55/manager/0.log"
Oct 30 20:29:27.970994 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:29:27.970967    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5d5b686b-r9n2j_24780d87-6614-4bf9-a587-89e043a10f55/manager/0.log"
Oct 30 20:29:35.196192 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:29:35.196144    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:29:41.073077 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:29:41.073046    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"]
Oct 30 20:29:41.121971 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:29:41.121948    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3eea3505-4m4xc"]
Oct 30 20:29:42.894963 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:29:42.894936    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd1beb3-3f4a-4b61-81fb-e63046d05ff2" path="/var/lib/kubelet/pods/1cd1beb3-3f4a-4b61-81fb-e63046d05ff2/volumes"
Oct 30 20:29:54.206076 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:29:54.206041    4420 scope.go:122] "RemoveContainer" containerID="7df11fd06b8f6656c9f0cb0ac966b212c738e48dfd04bc21ff717cc1bab4cb69"
Oct 30 20:30:05.195108 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:30:05.195022    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:30:35.195077 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:30:35.195042    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:30:46.999751 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:30:46.999722    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5d5b686b-r9n2j_24780d87-6614-4bf9-a587-89e043a10f55/manager/0.log"
Oct 30 20:30:47.588241 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:30:47.588180    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6b5d5b686b-r9n2j_24780d87-6614-4bf9-a587-89e043a10f55/manager/0.log"
Oct 30 20:30:54.224996 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:30:54.224965    4420 scope.go:122] "RemoveContainer" containerID="964f0e26468c925894ef3ed3b1f31fa6d62e72e19f2bedab2c864d48a6a1d302"
Oct 30 20:30:54.233556 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:30:54.233536    4420 scope.go:122] "RemoveContainer" containerID="444550121ea3131bd9095cc2f7cc24917d1019d562057330ff98bb6c3dfbeaa6"
Oct 30 20:31:05.195439 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:31:05.195403    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:31:08.846020 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:08.845992    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n5gpj_d87fa4fa-f703-4fc3-ac9f-dc9a012bdf51/global-pull-secret-syncer/0.log"
Oct 30 20:31:11.697219 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:11.697190    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sdtv7_866690aa-7dfb-4e2b-a29b-740a13dcc42e/konnectivity-agent/0.log"
Oct 30 20:31:12.484076 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:12.484042    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-7.us-west-1.compute.internal_67762efed53c0ad4c56b95d172cd37b4/haproxy/0.log"
Oct 30 20:31:35.193700 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:31:35.193620    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:31:45.860262 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.860232    4420 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"]
Oct 30 20:31:45.860668 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.860656    4420 cpu_manager.go:414] "containerMap: removing container" podUID="e1e4275e-c435-4e96-928f-7069c3b2d4ba" containerName="maas-api"
Oct 30 20:31:45.860710 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.860668    4420 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e4275e-c435-4e96-928f-7069c3b2d4ba" containerName="maas-api"
Oct 30 20:31:45.860745 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.860736    4420 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e4275e-c435-4e96-928f-7069c3b2d4ba" containerName="maas-api"
Oct 30 20:31:45.863858 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.863840    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:45.878071 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.878046    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"]
Oct 30 20:31:45.933597 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.933576    4420 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n79m4\"/\"default-dockercfg-jfgg8\""
Oct 30 20:31:45.933701 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.933615    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n79m4\"/\"openshift-service-ca.crt\""
Oct 30 20:31:45.933701 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.933642    4420 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n79m4\"/\"kube-root-ca.crt\""
Oct 30 20:31:45.997870 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.997838    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-lib-modules\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:45.998039 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.998021    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-podres\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:45.998106 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.998091    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-proc\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:45.998160 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.998123    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbkw\" (UniqueName: \"kubernetes.io/projected/c0818a1d-95c4-4148-ae75-912fa48957cb-kube-api-access-ljbkw\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:45.998308 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:45.998289    4420 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-sys\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.099541 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.099517    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-podres\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.099629 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.099551    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-proc\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.099629 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.099570    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbkw\" (UniqueName: \"kubernetes.io/projected/c0818a1d-95c4-4148-ae75-912fa48957cb-kube-api-access-ljbkw\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.099629 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.099608    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-sys\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.099766 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.099638    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-proc\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.099766 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.099641    4420 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-lib-modules\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.099766 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.099685    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-sys\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.099766 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.099638    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-podres\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.099965 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.099790    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c0818a1d-95c4-4148-ae75-912fa48957cb-lib-modules\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.155840 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.155786    4420 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbkw\" (UniqueName: \"kubernetes.io/projected/c0818a1d-95c4-4148-ae75-912fa48957cb-kube-api-access-ljbkw\") pod \"perf-node-gather-daemonset-fl472\" (UID: \"c0818a1d-95c4-4148-ae75-912fa48957cb\") " pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.176286 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.176263    4420 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:46.544328 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.544249    4420 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"]
Oct 30 20:31:46.546832 ip-10-0-133-7 kubenswrapper[4420]: W1030 20:31:46.546792    4420 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc0818a1d_95c4_4148_ae75_912fa48957cb.slice/crio-53db7a9bd64c0727021badb3ee5bef047ec98abbb0b2ad54abc29f93ecc7b34a WatchSource:0}: Error finding container 53db7a9bd64c0727021badb3ee5bef047ec98abbb0b2ad54abc29f93ecc7b34a: Status 404 returned error can't find the container with id 53db7a9bd64c0727021badb3ee5bef047ec98abbb0b2ad54abc29f93ecc7b34a
Oct 30 20:31:46.611729 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:46.611703    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472" event={"ID":"c0818a1d-95c4-4148-ae75-912fa48957cb","Type":"ContainerStarted","Data":"53db7a9bd64c0727021badb3ee5bef047ec98abbb0b2ad54abc29f93ecc7b34a"}
Oct 30 20:31:47.627046 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:47.627006    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472" event={"ID":"c0818a1d-95c4-4148-ae75-912fa48957cb","Type":"ContainerStarted","Data":"257550ce42ebb93d450db16370880f134539b9e0c4b983b61c0cb8443594f0fb"}
Oct 30 20:31:47.627580 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:47.627149    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:47.750283 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:47.750239    4420 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472" podStartSLOduration=2.750226616 podStartE2EDuration="2.750226616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:47.748192642 +0000 UTC m=+4554.959409786" watchObservedRunningTime="2026-10-30 20:31:47.750226616 +0000 UTC m=+4554.961443759"
Oct 30 20:31:52.417988 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:52.417957    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"]
Oct 30 20:31:52.418386 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:52.418139    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" podUID="ca4ca4ee-283d-4613-8d97-09478295784d" containerName="istio-proxy" containerID="cri-o://4f6f5d2f0d1f3ad62bb179d49d09bcec881bd59709f6f351ac32c0377f171cfc" gracePeriod=30
Oct 30 20:31:52.468363 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:52.468339    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"]
Oct 30 20:31:52.468551 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:52.468516    4420 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" containerName="istio-proxy" containerID="cri-o://b6963a6d9c775339a268c2b7177c996fa08505f9c700ee11f4c5ad2be4090672" gracePeriod=30
Oct 30 20:31:53.640747 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:53.640716    4420 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n79m4/perf-node-gather-daemonset-fl472"
Oct 30 20:31:57.680990 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.680955    4420 generic.go:356] "Generic (PLEG): container finished" podID="442ef4a0-7909-42c2-8567-8d7555361aee" containerID="b6963a6d9c775339a268c2b7177c996fa08505f9c700ee11f4c5ad2be4090672" exitCode=0
Oct 30 20:31:57.681419 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.681027    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" event={"ID":"442ef4a0-7909-42c2-8567-8d7555361aee","Type":"ContainerDied","Data":"b6963a6d9c775339a268c2b7177c996fa08505f9c700ee11f4c5ad2be4090672"}
Oct 30 20:31:57.682970 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.682948    4420 generic.go:356] "Generic (PLEG): container finished" podID="ca4ca4ee-283d-4613-8d97-09478295784d" containerID="4f6f5d2f0d1f3ad62bb179d49d09bcec881bd59709f6f351ac32c0377f171cfc" exitCode=0
Oct 30 20:31:57.683094 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.682982    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" event={"ID":"ca4ca4ee-283d-4613-8d97-09478295784d","Type":"ContainerDied","Data":"4f6f5d2f0d1f3ad62bb179d49d09bcec881bd59709f6f351ac32c0377f171cfc"}
Oct 30 20:31:57.725051 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.725028    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:31:57.744952 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.744929    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:31:57.796734 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.796714    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca4ca4ee-283d-4613-8d97-09478295784d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4ca4ee-283d-4613-8d97-09478295784d-istio-podinfo\") pod \"ca4ca4ee-283d-4613-8d97-09478295784d\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") "
Oct 30 20:31:57.796895 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.796761    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-certs\") pod \"ca4ca4ee-283d-4613-8d97-09478295784d\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") "
Oct 30 20:31:57.796895 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.796841    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-socket\") pod \"ca4ca4ee-283d-4613-8d97-09478295784d\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") "
Oct 30 20:31:57.796895 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.796871    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-istio-token\" (UniqueName: \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-istio-token\") pod \"ca4ca4ee-283d-4613-8d97-09478295784d\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") "
Oct 30 20:31:57.797067 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.796912    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca4ca4ee-283d-4613-8d97-09478295784d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4ca4ee-283d-4613-8d97-09478295784d-istiod-ca-cert\") pod \"ca4ca4ee-283d-4613-8d97-09478295784d\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") "
Oct 30 20:31:57.797067 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.796987    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-credential-socket\") pod \"ca4ca4ee-283d-4613-8d97-09478295784d\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") "
Oct 30 20:31:57.797067 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797045    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-kube-api-access-4nz8v\" (UniqueName: \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-kube-api-access-4nz8v\") pod \"ca4ca4ee-283d-4613-8d97-09478295784d\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") "
Oct 30 20:31:57.797067 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797056    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-certs" pod "ca4ca4ee-283d-4613-8d97-09478295784d" (UID: "ca4ca4ee-283d-4613-8d97-09478295784d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.797273 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797072    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-socket" pod "ca4ca4ee-283d-4613-8d97-09478295784d" (UID: "ca4ca4ee-283d-4613-8d97-09478295784d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.797273 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797107    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-data\") pod \"ca4ca4ee-283d-4613-8d97-09478295784d\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") "
Oct 30 20:31:57.797273 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797159    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-envoy\") pod \"ca4ca4ee-283d-4613-8d97-09478295784d\" (UID: \"ca4ca4ee-283d-4613-8d97-09478295784d\") "
Oct 30 20:31:57.797273 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797207    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-credential-socket" pod "ca4ca4ee-283d-4613-8d97-09478295784d" (UID: "ca4ca4ee-283d-4613-8d97-09478295784d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.797464 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797337    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-data" pod "ca4ca4ee-283d-4613-8d97-09478295784d" (UID: "ca4ca4ee-283d-4613-8d97-09478295784d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.797464 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797401    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4ca4ee-283d-4613-8d97-09478295784d-istiod-ca-cert" pod "ca4ca4ee-283d-4613-8d97-09478295784d" (UID: "ca4ca4ee-283d-4613-8d97-09478295784d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:57.797536 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797520    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.797601 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797541    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.797601 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797554    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.797601 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797571    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.797601 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.797586    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4ca4ee-283d-4613-8d97-09478295784d-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.799060 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.799038    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca4ca4ee-283d-4613-8d97-09478295784d-istio-podinfo" pod "ca4ca4ee-283d-4613-8d97-09478295784d" (UID: "ca4ca4ee-283d-4613-8d97-09478295784d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:31:57.799162 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.799085    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-istio-token" pod "ca4ca4ee-283d-4613-8d97-09478295784d" (UID: "ca4ca4ee-283d-4613-8d97-09478295784d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:57.799329 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.799309    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-kube-api-access-4nz8v" pod "ca4ca4ee-283d-4613-8d97-09478295784d" (UID: "ca4ca4ee-283d-4613-8d97-09478295784d"). InnerVolumeSpecName "kube-api-access-4nz8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:57.799387 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.799356    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-envoy" pod "ca4ca4ee-283d-4613-8d97-09478295784d" (UID: "ca4ca4ee-283d-4613-8d97-09478295784d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.898726 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.898700    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-certs\") pod \"442ef4a0-7909-42c2-8567-8d7555361aee\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") "
Oct 30 20:31:57.898877 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.898748    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-istio-token\" (UniqueName: \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-istio-token\") pod \"442ef4a0-7909-42c2-8567-8d7555361aee\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") "
Oct 30 20:31:57.898877 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.898778    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-data\") pod \"442ef4a0-7909-42c2-8567-8d7555361aee\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") "
Oct 30 20:31:57.898877 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.898806    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-credential-socket\") pod \"442ef4a0-7909-42c2-8567-8d7555361aee\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") "
Oct 30 20:31:57.898877 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.898867    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/442ef4a0-7909-42c2-8567-8d7555361aee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442ef4a0-7909-42c2-8567-8d7555361aee-istiod-ca-cert\") pod \"442ef4a0-7909-42c2-8567-8d7555361aee\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") "
Oct 30 20:31:57.899124 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.898906    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/442ef4a0-7909-42c2-8567-8d7555361aee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442ef4a0-7909-42c2-8567-8d7555361aee-istio-podinfo\") pod \"442ef4a0-7909-42c2-8567-8d7555361aee\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") "
Oct 30 20:31:57.899124 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.898952    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-certs" pod "442ef4a0-7909-42c2-8567-8d7555361aee" (UID: "442ef4a0-7909-42c2-8567-8d7555361aee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.899124 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.898975    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-envoy\") pod \"442ef4a0-7909-42c2-8567-8d7555361aee\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") "
Oct 30 20:31:57.899124 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899010    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-socket\") pod \"442ef4a0-7909-42c2-8567-8d7555361aee\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") "
Oct 30 20:31:57.899124 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899067    4420 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-kube-api-access-mgwcn\" (UniqueName: \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-kube-api-access-mgwcn\") pod \"442ef4a0-7909-42c2-8567-8d7555361aee\" (UID: \"442ef4a0-7909-42c2-8567-8d7555361aee\") "
Oct 30 20:31:57.899443 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899199    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/442ef4a0-7909-42c2-8567-8d7555361aee-istiod-ca-cert" pod "442ef4a0-7909-42c2-8567-8d7555361aee" (UID: "442ef4a0-7909-42c2-8567-8d7555361aee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 20:31:57.899443 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899352    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4ca4ee-283d-4613-8d97-09478295784d-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.899443 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899370    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4ca4ee-283d-4613-8d97-09478295784d-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.899443 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899385    4420 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-certs\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.899443 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899400    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.899443 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899414    4420 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/442ef4a0-7909-42c2-8567-8d7555361aee-istiod-ca-cert\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.899443 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899428    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nz8v\" (UniqueName: \"kubernetes.io/projected/ca4ca4ee-283d-4613-8d97-09478295784d-kube-api-access-4nz8v\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:57.899844 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899802    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-socket" pod "442ef4a0-7909-42c2-8567-8d7555361aee" (UID: "442ef4a0-7909-42c2-8567-8d7555361aee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.899896 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899868    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-credential-socket" pod "442ef4a0-7909-42c2-8567-8d7555361aee" (UID: "442ef4a0-7909-42c2-8567-8d7555361aee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.899936 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.899894    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-data" pod "442ef4a0-7909-42c2-8567-8d7555361aee" (UID: "442ef4a0-7909-42c2-8567-8d7555361aee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.901107 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.901081    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-istio-token" pod "442ef4a0-7909-42c2-8567-8d7555361aee" (UID: "442ef4a0-7909-42c2-8567-8d7555361aee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:57.901665 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.901645    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-envoy" pod "442ef4a0-7909-42c2-8567-8d7555361aee" (UID: "442ef4a0-7909-42c2-8567-8d7555361aee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 20:31:57.901665 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.901656    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-kube-api-access-mgwcn" pod "442ef4a0-7909-42c2-8567-8d7555361aee" (UID: "442ef4a0-7909-42c2-8567-8d7555361aee"). InnerVolumeSpecName "kube-api-access-mgwcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 20:31:57.902315 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:57.902296    4420 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/442ef4a0-7909-42c2-8567-8d7555361aee-istio-podinfo" pod "442ef4a0-7909-42c2-8567-8d7555361aee" (UID: "442ef4a0-7909-42c2-8567-8d7555361aee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 20:31:58.000436 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.000368    4420 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-data\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:58.000436 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.000400    4420 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-credential-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:58.000436 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.000419    4420 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/442ef4a0-7909-42c2-8567-8d7555361aee-istio-podinfo\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:58.000436 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.000434    4420 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-istio-envoy\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:58.000688 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.000447    4420 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/442ef4a0-7909-42c2-8567-8d7555361aee-workload-socket\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:58.000688 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.000461    4420 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgwcn\" (UniqueName: \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-kube-api-access-mgwcn\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:58.000688 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.000474    4420 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/442ef4a0-7909-42c2-8567-8d7555361aee-istio-token\") on node \"ip-10-0-133-7.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 20:31:58.688950 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.688910    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb" event={"ID":"ca4ca4ee-283d-4613-8d97-09478295784d","Type":"ContainerDied","Data":"456d5a961953e1e5baa8f663229186e5a63d1ba91039f7ab55abcbeb8e9802fb"}
Oct 30 20:31:58.688950 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.688945    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"
Oct 30 20:31:58.689421 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.688982    4420 scope.go:122] "RemoveContainer" containerID="4f6f5d2f0d1f3ad62bb179d49d09bcec881bd59709f6f351ac32c0377f171cfc"
Oct 30 20:31:58.690473 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.690441    4420 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj" event={"ID":"442ef4a0-7909-42c2-8567-8d7555361aee","Type":"ContainerDied","Data":"f28f9536b15d44898b61bf31197c0b710da91f58865510b7b5522432e025c337"}
Oct 30 20:31:58.690473 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.690462    4420 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"
Oct 30 20:31:58.701787 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.701682    4420 scope.go:122] "RemoveContainer" containerID="b6963a6d9c775339a268c2b7177c996fa08505f9c700ee11f4c5ad2be4090672"
Oct 30 20:31:58.851029 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.850996    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"]
Oct 30 20:31:58.900320 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:58.900289    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-8r4gb"]
Oct 30 20:31:59.010055 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:59.009991    4420 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"]
Oct 30 20:31:59.012598 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:31:59.012579    4420 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-748lj"]
Oct 30 20:32:00.893966 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:32:00.893938    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="442ef4a0-7909-42c2-8567-8d7555361aee" path="/var/lib/kubelet/pods/442ef4a0-7909-42c2-8567-8d7555361aee/volumes"
Oct 30 20:32:00.894360 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:32:00.894347    4420 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4ca4ee-283d-4613-8d97-09478295784d" path="/var/lib/kubelet/pods/ca4ca4ee-283d-4613-8d97-09478295784d/volumes"
Oct 30 20:32:05.195909 ip-10-0-133-7 kubenswrapper[4420]: E1030 20:32:05.195870    4420 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tqj9m" containerName="extractor"
Oct 30 20:32:16.971610 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:32:16.971566    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f6998b7b5-qbhn2_7a3667f5-8616-4e54-ba3e-f9a530059639/kube-auth-proxy/0.log"
Oct 30 20:32:17.402228 ip-10-0-133-7 kubenswrapper[4420]: I1030 20:32:17.402191    4420 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-cxf6d_94d386d8-73a7-4a53-8fec-2d70fcd9a2ef/istio-proxy/0.log"